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Abstract 


In  March  of  2001,  the  Software  Engineering  Institute  (SEI)  in  Pittsburgh,  PA,  hosted  a  work¬ 
shop  for  high  maturity  organizations  to  better  understand  practices  that  characterize  Capabil¬ 
ity  Maturity  Model®  for  Software  (Software  CMM®)  Level  4  and  5  organizations.  Topics  of 
discussion  included  practices  described  in  the  Software  CMM  as  well  as  other  practices  that 
have  a  significant  impact  in  mature  organizations.  Important  themes  included  statistical  proc¬ 
ess  control  for  software,  the  reliability  of  Level  4  and  5  assessments,  and  the  impact  of  the 
CMM  Integration®'^  effort.  Additional  topics  solicited  from  the  participants  included  meas¬ 
urement,  Six  Sigma,  Internet  speed  and  process  agility,  and  people  and  cultural  issues.  This 
report  contains  overviews  of  more  than  30  high  maturity  organizations  and  the  various  work¬ 
ing  group  reports  from  the  workshop. 


®  Capability  Maturity  Model  and  CMM  are  registered  in  the  U.S.  Patent  and  Trademark  Office. 
CMM  Integration  is  a  service  mark  of  Carnegie  Mellon  University. 
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1  Introduction 


A  workshop  for  high  maturity  organizations  was  held  on  March  27-29,  2001,  hosted  by  the 
Software  Engineering  Institute  (SEI)  in  Pittsburgh,  PA.  The  purpose  of  this  workshop  was  to 
better  understand  practices  that  characterize  the  Capability  Maturity  Model®  for  Software 
(Software  CMM®)  Level  4  and  5  organizations.  This  workshop  was  by  invitation  only.  The 
SEI  invited  representatives  from  all  known  Level  4  and  5  organizations  and  Lead  Assessors/ 
Evaluators  who  had  reported  assessing  a  Level  4  or  5  organization.  There  were  48  partici¬ 
pants,  representing  35  high  maturity  organizations.  The  individuals  participating  are  listed  in 
Appendix  A.  The  organizations  represented  are  listed  in  Appendix  B. 

Previous  high  maturity  workshops  were  held  in  1996  [Paulk  99]  and  November  1999  [Paulk 
00a].  Previous  surveys  of  high  maturity  organizations  were  held  in  1998  [Paulk  99]  and  1999 
[Paulk  00b]. 

Topics  of  discussion  included  both  practices  described  in  the  Software  CMM  and  other  prac¬ 
tices  that  have  a  significant  impact  in  mature  organizations.  Themes  that  were  anticipated  to 
be  important  to  the  workshop  participants  included  statistical  process  control  for  software, 
the  reliability  of  Level  4  and  5  assessments,  and  the  impact  of  the  CMM  Integration  effort. 
Additional  topics  solicited  from  the  participants  included  measurement,  Six  Sigma,  Internet 
speed  and  process  agility,  and  people  and  cultural  issues.  This  report  contains  overviews  of 
more  than  30  high  maturity  organizations  and  the  various  working  group  reports. 

The  workshop  began  with  a  welcome  by  Clyde  Chittister,  Chief  Operations  Officer  of  the 
SEI.  This  was  followed  by  an  overview  of  the  workshop  agenda.  The  proposed  working 
group  sessions  were  revised  by  the  attendees  to  make  the  most  effective  use  of  the  time  avail¬ 
able. 

A  survey  on  high  maturity  practices  was  distributed  in  February  2001  to  all  known  Level  4 
and  5  organizations^  Participants  in  the  workshop  were  briefed  on  the  preliminary  results  of 
that  survey  to  inspire  discussion  within  the  working  groups.  For  representatives  from  Level  4 
and  5  organizations,  completing  the  survey  was  a  prerequisite  for  attending  the  workshop,  as 
was  providing  an  organizational  summary  for  this  report.  Lead  Assessors/  Evaluators  were 
asked  to  provide  a  working  paper  on  their  insights  into  high  maturity  practices. 


®  Capability  Maturity  Model  and  CMM  are  registered  in  the  U.S.  Patent  and  Trademark  Office. 

^  Paulk,  Mark  C.,  Goldenson,  Dennis,  and  White,  David  M.  The  2001  Survey  of  High  Maturity  Organi¬ 
zations  (CMU/SEI-2001-SR-013). 
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Working  group  sessions  were  run  as  afternoon  plus  morning  discussions,  with  briefings  on 
working  group  conclusions  after  lunch.  A  general  workshop  debriefing  was  held  on  the  last 
day.  Steve  Cross,  Director  of  the  SEI,  closed  the  workshop  by  thanking  participants. 

The  high  maturity  organizations  invited  were  identified  as  the  result  of  appraisals  that  were 
conformant  with  the  CMM  Appraisal  Framework  (CAF).  These  are  usually  CMM-based  as¬ 
sessments  for  internal  process  improvement  (CB A IPI),  or,  less  frequently,  software  capabil¬ 
ity  evaluations  (SCE).  In  both  of  these  methods,  the  appraisal  team  should  be  led  by,  respec¬ 
tively,  an  SEI-authorized  Lead  Assessor  or  Lead  Evaluator.  There  are  also  a  few  instances  of 
CAF-conformant  appraisals  in  the  list,  where  the  appraisal  method  has  been  reviewed  and 
approved  by  the  SEI.  Appendix  B  notes  those  cases  where  the  appraisal  was  not  a  CBA  IPI. 
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2  Preliminary  Results  of  the  2001  Survey 
of  High  Maturity  Practices 


The  preliminary  results  of  the  2001  survey  of  high  maturity  practices  briefed  to  the  workshop 
participants  were  based  on  41  responses  from  132  organizations  reported  as  being  assessed  or 
evaluated  at  Level  4  or  5.  The  preliminary  results  are  summarized  below;  the  final  results  will 
be  published  as  an  SEI  special  report^. 

According  to  the  preliminary  results,  high  maturity  organizations  typically: 

•  have  an  independent  software  quality  assurance  (SQA)  organization 

•  use  domain-specific  software  architectures 

•  have  a  centralized  measurement  program 

•  have  required  training  in  management  skills 

Most  high  maturity  organizations: 

•  are  ISO  9001  certified 

•  have  a  Total  Quality  Management  (TQM)  program  for  the  assessed  organization  or  some 
higher  corporate  level 

•  embed  SQA  in  the  process 

•  use  incremental  or  evolutionary  life  cycles 

•  do  user  interface  prototyping 

•  have  independent  test  groups 

•  measure  code  coverage  of  testing 

•  have  product  lines  or  families 

•  do  other  forms  of  systematic  reuse 

•  use  Pareto  analyses 

•  use  control  charts  in  code,  test,  design,  and  requirements 

•  have  required  training  in  software  engineering  skills,  team  building,  domain  knowledge, 
interpersonal  skills,  and  change  management 

^  Paulk,  Mark  C.,  Goldenson,  Dennis,  and  White,  David  M.  The  2Q0I  Survey  of  High  Maturity  Organi¬ 
zations  (CMU/SEI-2001-SR-013). 
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Many  high  maturity  organizations: 


•  use  Balanced  Scorecard 

•  use  CMM  Integration^^  (CMMI®^’) 

•  use  Six  Sigma 

•  use  the  People  CMM 

•  use  Delphi  methods  for  estimating 

•  use  parametric  cost  models 

•  use  chief  architects  and  chief  engineers 

•  do  integrated  process  and  product  development 

•  use  earned  value 

•  use  defect  prediction,  reliability,  and/or  release  readiness  models 

•  use  formal  methods 

•  use  cost  of  quality 

•  use  orthogonal  defect  classification  or  other  defect  taxonomies 

•  use  control  charts  in  operations 

•  use  structured  English  (or  another  natural  language)  for  process  definition 

•  use  ETVX  (entry  criteria,  task,  verification,  and  exit  criteria)  process  modeling  notation 

•  have  a  formal  mentoring  program 

•  have  required  training  in  meeting  management 

Some  high  maturity  organizations: 

•  use  ISO/IEC  12207  (Software  Life  Cycle  Processes) 

•  use  Malcolm  Baldrige  National  Quality  Award  criteria 

•  use  Systems  Engineering  CMM 

•  use  ISO/IEC  15504  (Software  Process  Assessment) 

•  use  EIA/IS  731  (Systems  Engineering  Capability  Model) 

•  use  critical  chain 

•  use  PSP^'^  (Personal  Software  Process^*^)  and/or  TSP^"^  (Team  Software  Process^*^) 

•  use  Quality  Function  Deployment 

•  use  regression  analysis 

•  use  analysis  of  variance 

CMM  Integration,  CMMI,  PSP,  Personal  Software  Process,  TSP,  and  Team  Software  Process  are 
service  marks  of  Carnegie  Mellon  University. 
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•  use  process  modeling 

•  use  confidence  intervals 

•  use  prediction  intervals 

•  use  hypothesis  testing 

•  do  designed  experiments 

•  do  quasi-experimental  design 

•  use  other  multivariate  techniques 

•  use  EITVOX  (entry  criteria,  inputs,  task,  verification,  outputs,  and  exit  criteria)  process 
modeling  notation 

•  use  IDEFO  (function  modeling  method) 

•  use  SADT  (structured  analysis  and  design  technique) 

•  have  required  training  in  principled  negotiation 

All  of  the  responding  organizations  began  their  software  process  improvement  programs  be¬ 
fore  1998.  Ten  of  the  respondents  were  DOD  or  other  government  contractors,  two  were  gov¬ 
ernment  agencies,  two  were  commercial  shrinkwrap  organizations,  and  14  were  custom  soft¬ 
ware  developers.  Most  of  the  respondents  build  embedded  systems,  many  build  real-time 
applications,  and  some  develop  management  information  systems. 
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3  Working  Papers  from  Lead  Assessors 


Lead  Assessors  and  Evaluators  were  asked  to  provide  a  working  paper  on  assessing  high  ma¬ 
turity  organizations.  A  template  was  provided  with  five  sections  suggested. 

First,  the  Lead  Assessors  were  asked  to  include  any  desired  supplemental  information,  e.g., 
an  ISO  9001  auditor,  an  ISO  15504  assessor,  etc. 

Second,  with  respect  to  assessing  high  maturity  organizations:  What  were  some  of  the  diffi¬ 
cult  questions  they  had  to  answer  in  determining  whether  an  organization  was  Level  4  or  5? 
How  were  the  questions  resolved?  Why  was  the  decision  made  as  it  was?  What  are  the  issues 
seen  for  reliable  and  consistent  high  maturity  assessments?  Should  Level  5  organizations 
continue  to  reassess  using  CMM-based  appraisals  for  internal  process  improvement  (CBA 
IPI)  or  the  standard  CMMI  assessment  method  for  process  improvement  (SCAMPI^’^)  every 
2-3  years?  Are  there  more  effective  assessment  methods  for  high  maturity  organizations? 

Third,  in  characterizing  high  maturity:  What  are  some  of  the  things  that  characterize  high 
maturity,  i.e.,  a  low  maturity  organization  would  not  be  expected  to  do  this,  but  it  is  an  impor¬ 
tant  contributor  to  high  maturity  capability?  What  practices  are  high  maturity  organizations 
abandoning,  or  radically  changing,  as  they  move  to  Levels  4  and  5?  For  example,  are  there 
measures  that  might  be  useful  at  Level  2  that  are  either  abandoned  or  significantly  modified 
in  moving  to  Levels  4  and  5?  What  are  some  of  the  people  issues  that  high  maturity  organiza¬ 
tions  have  to  deal  with  that  may  be  different  or  addressed  differently?  Is  there  anything  spe¬ 
cial  about  team  building?  About  change  management?  About  skills  building?  Formal  mentor¬ 
ing  programs?  Extensive  induction  training  for  new  hires?  Is  turnover  an  issue?  Growth?  Has 
employee  morale  improved  as  a  result  of  the  process  improvement  activities? 

Fourth,  with  respect  to  continuing  improvement:  What  are  the  priorities  that  high  maturity 
organizations  need  to  work  on  next?  What  are  the  improvement  objectives?  The  practices  that 
they  are  adopting  or  refining?  What  are  the  biggest  barriers  they  are  currently  facing?  Are 
there  other  models  or  standards  they  are  using  (or  moving  to)? 

Fifth,  to  summarize:  What  does  it  mean  to  be  maturity  level  4  or  5?  What  is  different?  What 
are  the  effects  on  the  organization? 

Only  one  of  the  participating  Lead  Assessors  provided  the  working  paper  requested.  Several 
of  the  participating  Lead  Assessors  did  so  as  representatives  of  their  organization  and  may 

SCAMPI  is  a  service  mark  of  Carnegie  Mellon  University. 
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have  provided  feedback  on  the  issues  above  as  part  of  the  organizational  summaries  in  the 
next  section  of  this  report.  The  following  section  in  this  chapter  of  the  report  was  written  by 
the  Lead  Assessor  and  has  been  lightly  edited. 

3.1  Judah  Mogilensky,  Process  Enhancement 
Partners,  Inc. 

In  my  paper  for  the  2001  SEPG  Conference,  entitled  “Behavioral  Clues  to  Organizational 
Process  Maturity,”  I  described  my  approach  to  using  observable  behaviors  of  managers,  par¬ 
ticipants,  and  assessment  team  members  as  a  way  to  increase  or  decrease  confidence  in  as¬ 
sessment  results.  Specifically,  I  discussed  observable  behaviors  that  help  confirm  that  an  or¬ 
ganization  really  deserves  a  rating  of  Level  4  or  Level  5. 

The  perspective  that  leads  to  this  approach  comes  from  my  training  in  the  family  therapy 
techniques  of  Virginia  Satir.  In  family  therapy,  everything  is  relevant  data,  not  just  the  tran¬ 
scripts  of  the  “official  sessions.”  Examples  of  other  types  of  data  include: 

•  who  first  calls  the  therapist;  what  do  they  say 

•  who  shows  up  for  sessions 

•  in  what  order  do  they  enter  the  room 

•  where  do  they  sit,  what  body  postures  do  they  assume 

•  who  calls  between  sessions,  how  are  commitments  for  future  sessions  made 

All  of  these  items  are  considered  relevant  data  because  they  all  demonstrate  and  reveal  the 
relationships  and  interactions  going  on  within  the  family.  In  the  same  sense,  people  in  the 
software  process  improvement  community  have  long  recognized  that  maturity  levels  are  not 
just  unconnected  groups  of  practices,  but  they  are  cultural  patterns  that  extend  across  the  or¬ 
ganization.  (As  a  aside,  one  of  my  concerns  with  the  continuous  representation  used  in  the 
CMMI  models  is  precisely  that  it  does  encourage  viewing  process  areas  as  separate,  uncon¬ 
nected,  and  independent,  not  as  part  of  an  organization-wide  pattern  of  behavior.) 

In  my  work  with  organizations,  especially  performing  assessments  based  on  the  Software 
CMM,  I  began  to  notice  recognizable  behavior  patterns  that  characterized  the  different  matur¬ 
ity  levels.  I  realized  that  these  patterns  could  be  used  informally  to  confirm,  or  to  raise  doubts 
about,  the  “official”  assessment  data.  This  is  not  to  suggest  that  formal  assessment  ratings 
should  be  based  on  anything  but  the  official  data,  i.e.,  maturity  questionnaires,  document  re¬ 
views,  interviews,  draft  findings  feedback,  etc.  However,  useful  confirming  (or  disconfirm- 
ing)  data  can  be  observed  from  several  sources,  including: 

•  management  behavior  during  contracting  phase  and  delivery  phase 

•  participant  behavior  during  on-site  period 
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team  member  behavior  during  training  and  on-site  period 


Rather  than  discuss  my  indicators  of  low  (Level  1)  and  moderate  (Levels  2  and  3)  maturity,  I 

will  restrict  myself  to  indicators  of  high  (Levels  4  and  5)  maturity  in  this  section  of  the  report. 

Indicators  of  high  maturity  derived  from  observing  management  behavior  include: 

•  The  organization  has  collected  duration  and  effort  data  from  previous  assessments.  This 
information  is  offered  to  the  Lead  Assessor,  with  the  clear  expectation  that  it  will  be  used 
in  planning  the  current  assessment. 

•  Senior  managers  (including  senior  management  sponsors)  are  generally  not  included  in 
interviews  as  assessment  participants.  In  planning  the  current  assessment,  organization 
members  point  out  that  senior  managers  are  so  involved  in  the  process  that  failing  to  in¬ 
clude  them  would  risk  missing  crucial  data  about  the  process.  They  may  or  may  not,  in 
the  end,  be  included,  but  there  is  substantial  discussion  about  including  them. 


Indicators  of  high  maturity  derived  from  observing  assessment  participant  behavior  include: 

•  Across  the  different  interview  sessions,  there  is  widespread  awareness  among  the  partici¬ 
pants  of  what  measurements  are  used  on  projects,  and  how.  When  invited  to  bring  mate¬ 
rials  that  they  find  helpful  to  the  interview  sessions,  several  participants  bring  tables  and 
charts  with  them,  which  they  can  explain  in  detail. 

•  In  virtually  every  discussion  group,  participants  tell  new  stories  about  process  improve¬ 
ment  suggestions  that  were  made  and  implemented.  There  may  be  some  duplication  of 
the  examples  cited,  but  there  are  enough  instances  of  process  improvement  suggestions 
accepted  that  almost  all  the  ones  cited  are  new. 

•  During  at  least  one  group  discussion  session,  as  participants  from  different  projects  are 
discussing  some  problem  or  issue  they  have  encountered,  the  rough  equivalent  of  a  causal 
analysis  session  spontaneously  “breaks  out.”  Typically,  the  session  facilitator  must  inter¬ 
vene  to  stop  the  discussion  and  bring  the  discussion  session  back  to  the  planned  topics. 


Indicators  of  high  maturity  derived  from  observing  the  assessment  team  members  (assuming 

that  they  are  largely  drawn  from  the  organization  being  assessed)  include: 

•  The  typical  assessment  team  provided  by  a  high  maturity  organization  has  substantial 
prior  assessment  experience,  enough  to  put  an  experienced  team  member  on  every  mini¬ 
team.  Assessment  team  members  are  very  well  prepared  for  their  roles,  even  before  team 
training. 

•  The  team  routinely  completes  the  work  of  each  day  on  that  day.  While  catch-up  efforts 
are  done  if  they  are  needed,  such  efforts  are  rarely  needed. 

•  At  least  one  or  two  mini-teams  look  for  opportunities  to  get  ahead  of  the  scheduled  team 
work  plan  (e.g.,  by  starting  to  prepare  draft  findings  early). 
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•  My  only  instance  of  giving  a  team  a  scheduled  work  day  off  during  an  assessment  on-site 
period  was  for  a  Level  5  rated  organization,  when  the  team  completed  all  its  work  but  the 
senior  executive  sponsor  did  not  want  to  hold  the  Final  Findings  briefing  a  day  early. 


Again,  it  must  be  emphasized  that  no  single  instance  of  these,  or  other,  behaviors,  is  decisive. 
Rather,  what  I  am  looking  for  is  a  consistent  pattern  of  behavior  as  a  way  of  increasing  (or 
decreasing)  confidence  in  the  results  indicated  by  the  normal,  “official”  assessment  data. 
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4  Organizational  Summaries  of  High 
Maturity  Organizations 


Members  of  any  high  maturity  organization,  whether  they  planned  to  participate  in  the  work¬ 
shop  or  not,  were  invited  to  provide  an  organizational  summary.  These  descriptions  were  in¬ 
tended  to  be  brief,  on  the  order  of  1-3  pages,  although  summaries  were  allowed.  Organiza¬ 
tions  were  asked  to  provide  their  name,  city,  and  U.S.  state  or  country.  They  were  asked  to 
provide  the  summary  data  listed  in  the  table  below. 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor(s) 

Point  of  Contact 
Web  Page 

Size  of  the  Organization 
Typical  Program  Size 

Primary  Application  Domain(s) 


4  or  5 
month  year 

(or  Lead  Evaluators)  as  authorized  by  SEI  (or  company  if  CAF- 
conformant  assessment  rather  than  CB  A IPI) 

name  and  email  address  of  contact  person 
if  one  exists  for  the  organization 

number  of  software  professionals  (full-time  employees,  not  includ¬ 
ing  temporary  staff) 

number  of  people  per  typical  project 

number  of  lines  of  code  or  function  points  per  typical  program 
product  lines,  domain-specific  architectures,  etc. 


This  could  be  followed  by  any  desired  supplemental  information  about  the  organization  as  a 
whole.  Examples  from  the  previous  high  maturity  workshop  included  ISO  9001  certification, 
PSP/TSP  training,  history  of  the  software  process  improvement  program,  or  a  description  of 
assessment  variants  (CAF-conformant  corporate  assessment). 

They  were  asked  to  provide  retum-on-investment  (ROI)  and  improvement  trend  data.  This 
included  how  much  had  been  invested  in  software  process  improvement  (total  and  per  soft¬ 
ware  engineer)  and  what  kind  of  business  benefit  had  been  obtained  in  terms  of  cost,  sched¬ 
ule,  quality,  etc.,  i.e.,  the  return  on  investment.  It  also  included  the  primary  business  objec¬ 
tives  that  the  improvement  program  was  measured  against,  e.g.,  decrease  post  delivery 
defects  by  50%  within  one  year,  increase  customer  satisfaction  to  94%  within  one  year,  etc. 
Graphics  showing  improvement  trends  over  time  against  the  business  objectives  were  re¬ 
quested. 

They  were  asked  to  describe  the  barriers  to  achieving  high  maturity  that  they  had  encoun¬ 
tered.  These  barriers  could  be  process,  measurement,  cultural,  business  environment,  or  cus¬ 
tomer  relation.  The  intent  was  to  identify  the  things  that  had  to  be  done  differently  to  transi- 
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tion  to  Level  4  or  5  (or  the  issues  that  were  problems  before  but  became  dominant  problems 
that  had  to  be  solved  to  become  Level  4  or  5).  Particularly  useful  would  be  any  things  that  the 
organization  tried  and  abandoned  because  they  did  not  help. 

They  were  asked  to  describe  any  unique  or  distinguishing  practices  that  they  considered  char¬ 
acteristic  of  high  maturity,  i.e.,  a  low  maturity  organization  would  not  be  expected  to  do  this, 
but  it  is  an  important  contributor  to  the  high  maturity  capability  of  the  organization. 

They  were  asked  to  describe  any  people  and  cultural  issues  that  high  maturity  organizations 
have  to  deal  with  that  may  be  different  “  or  addressed  differently.  Is  there  anything  special 
about  team  building?  About  change  management?  About  skills  building?  Has  the  organiza¬ 
tion  established  a  formal  mentoring  program?  Extensive  induction  training  for  new  hires?  Is 
.turnover  an  issue?  Growth?  Has  employee  morale  improved  as  a  result  of  the  process  im¬ 
provement  activities? 


They  were  asked  to  describe  their  continuing  improvement  activities.  What  do  they  need  to 
work  on  next?  What  were  the  improvement  objectives?  What  were  the  practices  being 
adopted  or  refined?  What  were  the  biggest  barriers  the  organization  was  currently  facing? 
Did  they  plan  to  reassess  using  the  Software  CMM?  Were  there  other  models  or  standards 
they  were  using  (or  moving  to)? 


To  conclude,  they  were  asked  to  summarize  what  it  means  to  be  maturity  level  4  or  5.  What  is 
different?  What  are  the  effects  on  the  organization? 


The  following  sections  in  this  chapter  of  the  report  were  written  by  representatives  of  the 
various  high  maturity  organizations.  They  have  been  lightly  edited. 


4.1  Atos  Origin  india,  Mumbai,  India 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor(s) 

Point  of  Contact 
Size  of  the  Organization 
Typical  Program  Size 
Primary  Application  Domain(s) 


5 

November  2000 
Cyril  Dyer 

Darayus  S.  Desai,  darayusdesai@atosorigin.com 
--  350  at  the  time  of  the  assessment 

Average  project  size  is  ~15  persons  (ranges  from  5  to  40  persons) 
Commercial  custom-built  software 
Embedded  software 

Information  systems  software,  for  business  information 
Enterprise  resource  planning  (ERP)  packages 


Atos  Origin  India  is  part  of  Atos  Origin,  a  top  tier,  global  IT  consulting  and  services  com¬ 
pany,  with  operations  in  more  than  30  countries.  Atos  Origin  India  primarily  focuses  on  pro- 
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viding  the  full  range  of  software  engineering  and  ERP-related  services  to  clients  in  India  and 
across  the  globe. 

Atos  Origin  India  started  its  quality  journey  in  1993,  soon  after  its  inception,  and  achieved 
the  first  quality  nnilestone  with  the  ISO  9001  certificate  in  July  1994.  After  evaluating  various 
quality  models,  we  decided  to  adopt  the  Software  CMM.  An  informal  CMM  assessment  in 
January  1996  found  us  to  be  close  to  Level  3.  During  that  period  we  were  growing  very  rap¬ 
idly  and  undergoing  substantial  change  in  terms  of  the  service  portfolio  and  nature  of  busi¬ 
ness.  Consequently,  the  Quality  System  went  through  several  evolutions,  until  it  reached  a 
stage  where  it  had  to  be  completely  revamped.  This  meant  a  major  improvement  initiative, 
led  by  the  P&Q  (Processes  and  Quality)  function  with  help  from  a  number  of  task  forces  con¬ 
sisting  of  seasoned  practitioners. 

After  a  number  of  informal  mini  assessments  along  the  way,  we  underwent  a  formal  assess¬ 
ment  in  October  1999,  when  we  were  assessed  at  Software  CMM  Level  4,  followed  by  an¬ 
other  formal  assessment  in  November  2000  for  Level  5. 

4.1 .1  ROI  and  Improvement  Trend  Data 

While  we  have  realized  returns  in  various  areas,  one  of  the  major  benefits  has  been  the  con¬ 
stant  upward  trend  in  customer  satisfaction.  While  no  customer  rated  us  at  5  (on  a  scale  of  0 
to  5)  until  1997,  today  we  have  40%  of  the  customer  satisfaction  surveys  with  a  rating  of  5, 
and  we  have  built  strong  long-term  partnerships  with  our  customers. 

We  have  also  seen  a  significant  improvement  in  the  defect-free  deliveries  and  achievement  of 
planned  schedules. 

The  bottom  line  has  also  shown  a  positive  trend  over  these  years,  although  there  are  various 
other  factors  too,  besides  process  improvement,  that  have  contributed  to  that. 

4.1.2  Barriers  to  Achieving  High  Maturity 

Some  customers  have  a  misconception  about  processes  and  the  Software  CMM;  they  per¬ 
ceive  these  as  overheads,  both  in  terms  of  time  and  cost.  Further,  some  customers  themselves 
are  at  a  very  low  maturity  level  and  convincing  them  to  follow  at  least  some  basic  processes 
is  also  a  tough  job.  This  barrier  was  overcome  by  educating  and  convincing  them  about  the 
benefits  of  a  structured  process  approach  and  assuring  them  that  it  was  not  going  to  affect 
their  costs  and  schedules. 

Several  employees  also  felt  that  their  job  was  “software  engineering,”  and  Software  CMM  or 
ISO  did  not  seem  to  fit  into  their  scheme  of  things.  They  had  to  be  educated  and  convinced 
about  the  reasons  why  we  were  doing  this  and  the  benefits.  This  was  done  through  regular 
awareness  programs,  quality  forums  and  actual  case  studies.  Furthermore,  the  Quality  System 
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was  written  based  on  our  business  model  avoiding  any  Software  CMM  or  ISO  specific  ori¬ 
entation. 

Initially,  we  had  the  tendency  to  take  too  many  measurements,  many  of  which  were  not  being 
subsequently  used.  People  also  felt  that  there  were  too  many  forms.  So  these  were  reviewed 
critically  and  several  forms  were  merged,  and  some  forms  and  measurements  were  dropped 
to  avoid  duplication  and  remove  the  extraneous  ones. 

4.1.3  Unique  or  Distinguishing  Practices 

These  include 

•  high  level  of  senior  management  commitment — not  just  in  terms  of  statements  and  fund¬ 
ing,  but  also  demonstrated  by  actual  participation  in  process  improvement  activities 

•  usage  of  tools — preferably  ones  with  the  workflow  integrated  into  them 

•  higher  level  of  sharing  knowledge  and  experiences  across  the  organization  (not  just 
within  business  units) 

•  stronger  focus  on  proactive  improvements 

•  in-depth  understanding  of  our  processes 

4.1.4  People  and  Cultural  Issues 

Some  of  the  people  issues  have  already  been  described  in  the  section  on  barriers. 

Since  we  get  new  recruits  with  varied  backgrounds  from  diverse  organizations,  it  is  crucial  to 
get  them  into  the  Atos  Origin  quality  culture.  Specific  awareness  programs  and  refresher 
courses  are  held  regularly  to  build  and  sustain  this  culture.  Creating  process  action  teams 
with  people  across  different  business  units  helped  in  team  building,  as  well  as  greater  sharing 
of  experiences  across  the  entire  organization,  thus  reducing  “compartmentalization.” 

Involving  practitioners  in  process  improvement  initiatives  helped  build  a  better  sense  of 
ownership  and  disseminate  the  culture  further  into  the  organization. 

During  the  initial  stages  of  the  Software  CMM  initiative  there  was  a  certain  amount  of  resis¬ 
tance  and  the  morale  even  went  down  slightly,  but  as  people  started  to  see  the  results  and 
benefits  over  a  period  of  time,  it  improved  substantially. 

Software  CMM  does  not  address  people  issues  adequately,  and  that  gap  has  to  be  filled  in  by 
other  practices. 
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4.1.5  Continuing  Improvement 

We  are  currently  working  on  ISO  9001:2000  and  also  starting  with  the  People  CMM  initia¬ 
tive.  The  objective  is  to  excel  in  whatever  we  do;  not  just  to  achieve  business  goals  set  by 
ourselves,  but  also  perform  well  by  benchmarking  against  other  best-in-class  organizations. 

4.1.6  Summary 

Level  5  provides  the  basic  foundation  of  well-understood  processes  and  a  strong  culture  ori¬ 
ented  towards  process  and  quality  improvement.  We  intend  to  use  it  as  a  launching  pad,  for 
aiming  for  greater  heights  such  as  business  excellence,  and  ensuring  that  besides  achieving 
our  business  goals,  we  are  able  to  benchmark  ourselves  with  other  best-in-class  organiza¬ 
tions. 


Customer  satisfaction  has  gone  up  tremendously  and  so  has  repeat  business. 

Overall,  it  has  enhanced  our  image  greatly,  in  the  eyes  of  our  customers,  other  sister  organiza¬ 
tions,  as  well  as  our  employees,  who  feel  proud  to  be  part  of  a  Level  5  organization. 


4.2  Boeing’s  Reusable  Space  Systems  (RSS)  and 
Satellite  Programs  (SP)  in  Downey  and  Seai 
Beach,  CA 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor(s) 

Point  of  Contact 
Web  Page 


Size  of  the  Organization 
Typical  Program  Size 


Primary  Application  Domain(s) 


5 

October  1999 
JeffFacemire 
Andreas  R.  Felschow 

D.  Dillehunt,  Donald.Dillehunt@west.boeing.com 
Boeing  Space  &  Communications  Group  (overview): 
http://www.boeing.com/defense-space/sc-back/index.html 
Commercial  Information  Systems  (formerly  Satellite  Programs): 
http://www.boeing.eom/defense-space/sc-back/index.html#cis 
Reusable  Space  Systems: 

http://www.boeing.eom/defense-space/sc-back/index.html#rss 

350 

Number  of  people  per  typical  project  is  around  100 

Number  of  lines  of  code  or  function  points  per  typical  program  is 
about  500,000  SLOC 

Military  software,  adhering  to  DoD  standards,  and  systems  software, 
used  to  control  physical  devices 


In  1984,  Rockwell’s  Space  Systems  Division  (SSD)  became  a  technology  transfer  site  for 
Southern  California,  and  SSD’s  Software  Engineering  Directorate  was  formed.  A  Software 
Methods  group  was  created  in  1986,  a  precursor  to  the  Software  Engineering  Process  Group 
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(SEPG).  An  informal  self-assessment  was  performed  in  1989  and  again  in  1991.  At  this  time, 
SSD  had  projects  supplying  products  to  DoD,  NASA,  and  commercial  customers,  where  each 
customer  used  its  own  terms  and  required  different  reviews  and  products.  We  included  all 
customer  types  in  our  March  1992  self-assessment  (through  SEI  Software  CMM  Level  3).  In 
April  1992,  Rockwell  had  the  Software  Productivity  Consortium  (SPC),  an  authorized  SEI 
assessment  organization,  validate  revisions  to  our  improvement  plan,  and  our  CMM-based 
self-assessment. 

In  September  1994,  another  internal  process  improvement  appraisal  was  held  in  which  we 
evaluated  ourselves  as  meeting  the  criteria  for  a  Level  3  organization.  The  four  largest  soft¬ 
ware  projects  participated  in  this  appraisal,  sampling  our  entire  customer  base.  In  1995,  we 
received  our  ISO  9001  certification.  In  1997,  the  scope  of  our  Software  CMM  effort  was  ex¬ 
panded  to  five  projects  and  self-evaluation  through  Level  4  as  our  business  grew.  In  1998,  our 
division  Joined  The  Boeing  Company,  and  the  following  reorganization  effort  was  focused  on 
three  remaining  projects.  In  1999,  we  were  assessed  that  we  were  operating  at  an  SEI  Soft¬ 
ware  CMM  Level  5.  A  CBA IPI  was  performed  on  Reusable  Space  Systems  (RSS)  and  Satel¬ 
lite  Programs  (SP).  Three  projects  representing  over  70%  of  the  embedded  flight  software 
staff  were  evaluated.  We  are  the  third  and  largest  Boeing  organization  to  receive  a  Level  5 
rating. 

4.2.1  ROI  and  Improvement  Trend  Data 

We  were  involved  in  two  Software  Capability  Evaluations  and  one  Software  Development 
Capability  Evaluation,  the  results  of  which  contributed  to  multimillion-dollar  contract 
awards.  Additionally,  process  improvements  have  lowered  our  development  costs  as  shown  in 
Figure  1  (development  costs  are  normalized  to  the  year  1989’s  hours  /  SLOC  values).  We 
have  reduced  our  defect  rates  to  such  an  extent  that  one  project  has  had  no  defects  reported  in 
its  delivered  software  products  since  1997.  Our  investment  in  process  improvement  has  re¬ 
mained  stable  during  this  time,  with  an  average  of  nine  personnel  per  year  working  on  organ¬ 
izational  process  activities  from  1992  to  1999.  Additional  personnel  on  each  project  were  as¬ 
signed  project  process  activities. 


Figure  1:  Boeing  Process  Improvements  Lower  Development  Effort  per  Line  of  Code 
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Figure  2  represents  the  training  performed  on  a  yearly  basis.  Attendees  shown  in  the  chart 
reflect  the  number  of  personnel  attending  classes;  personnel  attending  multiple  classes  are 
counted  for  each  class  attended.  Note  that  more  training  is  required  each  time  the  organiza¬ 
tion  strives  to  reach  a  new  level.  Once  the  level  is  reached,  and  most  personnel  are  trained, 
training  drops  off  during  the  next  improvement  planning  cycle.  Level  3  was  attained  in  1994 
and  Level  5  appraised  in  October  1999. 


Figure  2:  Significant  Process  Training  Needed  To  Reach  Software  CMM  Level  5 

4.2.2  Barriers  to  Achieving  High  Maturity 

Level  4  required  additional  personnel  training  on  the  application  of  a  form  of  statistical  con¬ 
trol  process  to  be  used  across  the  organization.  Due  to  the  merger  of  Boeing  and  McDonnell 
Douglas,  we  were  able  to  leverage  pre-existing  McDonnell  Douglas  Malcolm  Baldrige  train¬ 
ing,  reducing  the  investment  cost  for  Level  4  training  activities. 

Being  a  Level  4  organization  means  projects  must  be  convinced  that  investing  in  this  type  of 
statistics  will  provide  benefits  over  the  course  of  the  program.  Having  been  a  Level  3  organi¬ 
zation  for  a  significant  time,  our  projects  saw  that  the  benefits  in  customer  satisfaction,  re¬ 
duced  errors,  and  on-time  deliveries  were  worth  the  project  management  and  peer  review  ef¬ 
fort  expended.  Level  4  was  a  leap  of  faith  for  projects  that  did  not  have  previous  experience 
to  provide  assurance  that  the  new  efforts  would  benefit  them.  The  organization  needs  to  pro¬ 
vide  support  to  assist  projects  until  the  benefits  are  realized  and  processes  internalized. 

There  are  added  project  costs  to  provide  data  if  it  is  a  test  project  for  a  new  technology  or 
process.  Acquiring  that  data  is  probably  one  of  the  hardest  obstacles  that  must  be  overcome  to 
become  a  Level  5  organization.  If  a  project  has  traveled  the  Software  CMM  path  from  a 
lower  level,  it  is  more  likely  that  it  may  feel  an  obligation  to  the  organization  for  benefits  de¬ 
rived  from  the  long  association.  Obtaining  data  from  new  and  especially  short-term  projects 
is  more  difficult  and  can  require  persistence  from  SEPG  personnel. 
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4.2.3  Unique  or  Distinguishing  Practices 

As  a  Level  5  organization,  we  provide  consistency  across  all  projects  regarding  availability  of 
tools  and  processes,  the  collection  of  metrics  through  our  integrated  tool  set,  and  project 
setup  and  defect  prevention  activities.  Continued  organizational  investment  in  mechanisms 
for  cross-project  dissemination  of  lessons  learned  also  is  a  critical  factor  in  maintaining  open 
lines  of  communication  between  projects. 

4.2.4  People  and  Cultural  Issues 

The  Space  Beach  Host  Engineering  Mentorship  Program  began  in  1990  and  has  had  more 
than  300  participants.  This  program  is  used  as  an  adjunct  to  the  more  formal  software  process 
training  program,  and  has  proven  to  be  very  valuable  in  transferring  skills  from  experts  to 
new  personnel.  The  goal  of  the  mentorship  program  is  to  capture  critical  skills  within  engi¬ 
neering  by  providing  training  in  meta-cognitive  skills  and  mentoring  strategies. 

4.2.5  Continuing  improvement 

Our  most  significant  challenge  is  business  consolidation,  with  the  associated  reorganization 
and  restructuring  of  our  financial  resources.  In  December  1999,  our  business  unit  from 
Downey,  CA,  was  consolidated  with  the  business  unit  at  a  former  McDonnell  Douglas  facil¬ 
ity  in  Huntington  Beach.  As  of  January  2001,  our  core  engineering  function  now  hosts  four 
separate  business  units:  Human  Space  Flight  and  Exploration  (HSF&E),  Expendable  Launch 
Systems  (ELS),  Integrated  Defense  Systems  (IDS),  and  Phantom  Works  (PW). 

These  groups  are  located  mainly  at  three  sites  in  California:  Huntington  Beach,  Seal  Beach, 
and  Long  Beach.  Our  near-term  challenge  continues  to  be  the  consolidation  of  McDonnell 
Douglas,  Rockwell,  and  Boeing  procedures  and  personnel  within  a  Boeing  structure,  includ¬ 
ing  a  reallocation  of  funding,  project  and  core  responsibilities,  and  personnel.  Additionally, 
the  president  of  Space  and  Communications  has  challenged  all  business  units  to  reach  Soft¬ 
ware  CMM  Level  5.  In  addition  to  achieving  higher  maturity  levels,  we  are  now  working  to 
address  changes  as  a  result  of  last  year’s  release  of  the  new  system  and  software  integrated 
CMM,  i.e.,  the  CMMI,  as  well  as  changes  associated  with  moving  from  ISO  9001  to  AS9100. 

4.2.6  Summary 

Being  a  Level  5  organization  means  investment  in  the  future.  Our  organization  funds  research 
into  new  processes  and  technologies,  verifies  usefulness,  and  provides  this  data  to  projects. 
Our  organization  recommends  best  practices.  As  projects,  business  thrusts,  and  customer 
bases  change,  our  process  needs  to  evolve  to  reflect  the  current  situation.  Our  motto  has  been 
“a  race  without  a  finish”  which  requires  our  continued  commitment  to  meet  the  needs  of  both 
our  organization’s  projects  and  their  customers  for  us  to  remain  successful.  Continued  in¬ 
vestment  and  vigilance  in  assessment  of  future  goals  is  needed  to  stay  at  the  top. 
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4.3  CG-Smith  Software,  Bangalore,  India 

5 

October  1999 
Richard.  F.  Storch 
Raghavendra  Swamy 
www.cg-smith.com 
260 

0.17  KSLOC-350  KSLOC 

Commercial  software  in  real-time  embedded  systems:  automotive 
electronics  and  aerospace,  data  communications  and  telecommunica¬ 
tions,  process  control  and  instrumentation,  medical  electronics,  lan¬ 
guage  processing,  and  electronic  design  automation  (EDA) 

4.3.1  ROI  and  Improvement  Trend  Data 

On  average,  productivity  has  tripled  in  the  last  five  years. 

4.3.2  Barriers  to  Achieving  High  Maturity 

Some  of  the  barriers  that  we  have  faced  include 

•  resistance  to  new  process  change 

•  middle  management  buy-in 

•  working  with  customers  who  are  at  different  process  maturity 

4.3.3  Unique  or  Distinguishing  Practices 

The  unique  practices  at  CG-Smith  (CGS)  that  characterize  the  high  maturity  of  the  organiza¬ 
tion  and  that  distinguish  CGS  from  low  maturity  organizations  are: 

•  CGS  has  a  proven  proprietary  software  development  methodology  that  has  delivered  out¬ 
standing  results  to  more  than  175  projects  executed.  The  methodology  is  based  on  the 
Uniphase  model. 

Uniphase:  the  process  model  that  is  followed  at  CGS.  Uniphase  is  composed  of  four  ba¬ 
sic  elements,  the  process,  the  screen,  the  store,  and  Management  and  Control  (M&C). 

Process:  defines  the  transformation  activities  of  the  Uniphase. 

Screen:  identifies  the  techniques  that  verify  and  validate  products  produced  by  the  proc¬ 
ess  elements. 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor(s) 

Point  of  Contact 
Web  Page 

Size  of  the  Organization 
Typical  Program  Size 
Primary  Application  Domain(s) 


CMU/SEI-2001-SR-014 


19 


Store:  identifies  the  input  and  output  products  of  the  software  process  element  as  well  as 
their  source  and  destination.  It  configures,  manages,  and  controls  the  software  products 
produced  by  the  process  element. 

Management  and  Control  (M&C):  identifies  the  resources  and  the  mechanisms  required 
to  monitor  and  control  the  process,  screen,  and  store  elements  within  the  Uniphase.  It  de¬ 
scribes  the  measurements  to  be  taken  and  the  reports  to  be  generated,  as  well  as  identifies 
all  persons  responsible  for  managing  the  Uniphase. 

•  SPN  (Structured  Process  Notation):  identifies  the  work  breakdown  structure  (WBS)  and 
is  used  to  define  the  transformation  activities  of  the  Uniphase  process  element.  It  identi¬ 
fies  the  sequence  of  activities  as  well  as  the  linkage  of  activities. 

•  Quantitative  Process  Management:  similar  to  how  the  CEO  of  the  company  decides  on 
the  performance  of  the  company  by  just  two  numbers,  i.e.,  debit  /  credit  and  profit  /  loss, 
projects  at  CGS  are  managed  effectively  and  more  predictably  only  by  six  numbers.  They 
are: 

-  Engineering  Effort  (WE):  total  engineering  effort  for  the  product. 

-  Rework  Effort  (RE):  total  rework  effort  for  the  product.  This  includes  rework  due  to 
additional  customer  requirements  and  defect  correction. 

“  Pre-Release  Defects  (DF):  defects  found  before  the  product  gets  stored. 

-  Post-Release  Defects  (DE):  defects  found  after  the  product  has  been  stored. 

“  Changes  from  Customer  (FC):  changes  initiated  to  incorporate  additional  require¬ 
ments  from  the  customer. 

“  Changes  Due  to  Defects  (UC):  Changes  initiated  due  to  post  release  defects. 

•  Process  automation:  a  high  level  of  process  automation  has  been  done  at  CGS.  The  tool 
consists  of  seven  modules: 

-  Process  Definition:  allows  the  user  to  define  the  WBS  for  the  project.  It  gives  the 
user  enough  flexibility  to  separate  a  huge  activity  or  integrate  two  small  ones,  divide 
the  project  as  per  the  products  that  are  to  be  delivered,  etc. 

“  Project  Management:  helps  manage  project  estimation,  project  tracking  and  analysis, 
and  organizational  performance  analysis. 

-  Defect  Management:  helps  manage  defect  recording,  defect  tracking  and  analysis, 
and  defect  prevention. 

-  Change  Management:  helps  in  version  management,  managing  change  requests,  and 
change  implementation  tracking  and  analysis. 

-  Knowledge  Management:  helps  to  manage  the  knowledge  capture  at  the  personal, 
project  team,  and  organizational  level.  The  knowledge  thus  captured  is  available  on¬ 
line  to  each  engineer  in  an  organization. 

“  Process  Improvement:  helps  to  record  process  suggestions  and  monitors  new  process 
implementation. 

-  Resource  Management:  helps  to  populate  and  maintain  human  resource  skill  reposi¬ 
tory,  logs  and  maintains  equipment  history,  and  matches  project  needs  with  suitable 
resources. 

This  unique  Process  Automation  tool  provides  process,  project,  and  knowledge  manage¬ 
ment  solutions.  It  focuses  on  the  intricacies  involved  in  process  definition,  project  estima- 
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tion  and  tracking,  management  of  defects  and  changes,  and  helps  build  a  continuously 
improving  organization. 

•  Process  Improvement:  CGS  has  a  very  active  process  improvement  program  in  place.  For 
the  last  five  years,  we  have  received  close  to  600  process  improvements  from  the  engi¬ 
neers  in  the  organization,  testimony  to  the  fact  that  CGS  is  a  learning  and  improving  or¬ 
ganization. 

4.3.4  People  and  Cultural  Issues 

We  have  developed  procedures  and  practices  in  line  with  the  requirements  of  the  People 
CMM,  which  addresses  issues  such  as  training,  mentoring,  skills  building,  and  career  growth. 
All  these  are  at  different  levels  of  institutionalization  in  the  company.  Training  has  been  iden¬ 
tified  for  all  levels  of  people  in  the  organization,  which  includes  training  on  process,  engi¬ 
neering,  domain,  and  technology.  Induction  training  for  new  recruits  is  very  comprehensive 
and  covers  all  areas  of  software  development  and  the  soft  skills  required.  Mentoring  is  being 
done  by  the  managers. 

Further  we  have  a  very  good  process  improvement  program  in  place  with  a  reward  mecha¬ 
nism  associated.  Employees  are  reaping  the  benefits,  and  the  morale  of  the  engineers  is  very 
high  as  they  see  a  distinct  difference  and  benefits  from  process  improvement.  However, 
growth  of  the  organization  calls  for  a  more  dedicated  and  focused  effort  to  sustain  the  above 
program. 

We  are  maintaining  a  very  comprehensive  skill  database,  which  details  the  complete  skills 
that  engineers  have  acquired  to  date,  including  the  competencies  and  expertise  that  he/she  has 
acquired  at  CGS.  The  skill  database  is  part  of  our  Process  Automation  tool,  which  gives  in¬ 
stant  information  to  the  line  managers  about  the  skills  acquired.  This  helps  managers  plan  for 
the  training  to  build  skills  of  the  engineers. 

4.3.5  Continuing  Improvement 

The  biggest  barrier  facing  us  is  with  respect  to  changing  the  attitude  of  people  towards  creat¬ 
ing  new  processes,  dismantling  processes,  or  changing  existing  processes,  as  the  tendency  is 
“If  it’s  working,  why  change  it?”  However,  CGS  management  has  strengthened  its  Process 
Improvement  initiative  by  substantially  enhancing  the  reward  program  and  also  has  tied  pro¬ 
cess  improvement  initiatives  to  the  employee  appraisal  system.  This  has  yielded  results  to  a 
greater  extent  in  changing  the  attitude  regarding  continuous  improvement.  Since  being  as¬ 
sessed  at  Level  5,  we  have  received  282  process  improvement  proposals. 

The  main  objective  of  improvement  has  been  to  improve  productivity,  reduce  delivered  de¬ 
fects,  and  reduce  schedule  slippage  further,  ultimately  to  achieve  our  goal  of  total  customer 
satisfaction.  This  desire  has  allowed  us  to  look  towards  CMMI  for  further  improvement. 

We  plan  to  reassess  using  the  Software  CMM  in  2001  or  assess  with  CMMI  in  2002. 
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4.3.6  Summary 

CGS  was  assessed  at  Level  5  in  October  1999.  CGS  has  gotten  exposure  worldwide  as  being 

one  of  the  few  companies  who  have  got  Level  5. 

Some  of  the  noticeable  differences  have  been  in  the  areas  of: 

•  Process  improvement — newer  process  and  improvement  to  the  existing  process  are  de¬ 
fined  resulting  in  better  software  development. 

•  Focus  on  customer  satisfaction — survey  results  have  helped  to  focus  on  issues  perceived 
by  the  customer  in  turn  helping  to  fine  tune  our  process  to  serve  customers  better. 

•  Control  of  project  schedule  overruns — because  of  well  laid  out  planning,  tracking  and 
working  with  anticipation,  project  schedule  overruns  have  been  controlled  to  a  greater 
extent. 

•  Reduction  in  delivered  defects — because  of  the  increased  focus  through  out  the  life  cycle 
of  project  development. 

•  Estimation  has  become  more  realistic  and  software  development  has  become  more  pre¬ 
dictable. 


4.4  Cognizant  Technology  Solutions,  Chennai,  India 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor(s) 
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Size  of  the  Organization 
Typical  Program  Size 


Primary  Application  Domain(s) 
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Emani  Sarathy,  esarathy@chn.cognizant.com 

http://www.cognizant.com 

3245 

Number  of  people  per  project  ranges  between  3-50,  depending  on  the 
type  of  project.  Projects  at  CTS  are  categorized  as  small  and  large 
projects  depending  on  the  duration  and  effort  required. 

Financial  services,  health  care,  information-defined  services,  diversi¬ 
fied,  insurance,  government,  hospitality,  IT  services,  retailer,  telecom, 
travels,  and  utilities. 


Cognizant’s  road  map  to  quality: 

•  ISO  9001  certified  in  1996 

•  assessed  at  Level  4  of  the  Software  CMM  in  December  1998 

•  re-certified  ISO  9001  in  March  1999 

•  assessed  at  Level  5  of  the  Software  CMM  in  September  2000 


The  following  are  some  of  the  key  initiatives  of  Cognizant  Academy: 
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•  Personal  Software  Process  (PSP) : 

-  Personal  Software  Process  course  offered  by  Carnegie  Mellon  University 

•  Cognizant  Certified  Professional  (CCP)  Program: 

-  introduced  the  concept  of  internal  certification  in  Cognizant  in  February  2000 

-  topics  were  finalized  based  on  the  competency  and  requirements  at  Cognizant  and 
wherever  external  certifications  were  not  available 

-  IBM  Mainframe,  IBM  Web  Technologies,  AS400,  S/W  Engineering,  Quality  and 
Process  Management,  Business  Development,  Client  Management 

•  Congregation  ©Cognizant: 

-  to  create  a  common  platform  for  Cognizant  Associates  to  showcase  best  practices, 
technologies,  and  tools 

4.4.1  ROI  and  Improvement  Trend  Data 

Improvements  in  effort,  schedule  variation,  and  the  rework  effort  have  resulted  in  achieving 
the  business  objectives  of  the  organization.  Effort  variation  %  has  reduced  from  21.67  to 
8.69.  Variation  spread  has  reduced  from  ±  14.74  to  ±  7.78.  Schedule  variation  %  has  reduced 
from  23.37  to  11.39.  Variation  spread  has  reduced  from  ±  13.51  to  ±  7.60.  Rework  %  has  re¬ 
duced  from  9. 19  %  to  5. 19  %  of  the  total  effort. 
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Figure  3:  Cognizant  Customer  Satisfaction 


As  shown  in  Figure  3,  customer  satisfaction  has  improved. 
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Figure  5:  Cognizant  Cost  of  Quality  Analysis  for  Cognizant 

Figure  5  illustrates  the  cost  of  quality  analysis  at  Cognizant,  while  Figure  6  compares  Cogni¬ 
zant  performance  with  industry  standards  as  published  by  Krasner. 
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Figure  6:  Industry  Norms  for  Cost  of  Quality  Compared  to  Cognizant 

4.4.2  Barriers  to  Achieving  High  Maturity 

The  following  were  found  to  be  difficult  while  transitioning  from  maturity  level  4  to  maturity 
level  5: 

•  Defect  Prevention:  creating  a  defect  database  from  over  150  projects  of  different  plat¬ 
forms  and  different  operational  models 

•  Technology  Change  Management:  quantitatively  measuring  the  impact  of  upcoming 
technologies 


4.4.3  Unique  or  Distinguishing  Practices 

Practices  related  to  people: 

•  training  on  domain  specific  areas 

•  practicing  3Ps:  Positive  attitude,  Passion,  Perseverance 

•  availability  of  pool  of  personnel  on  various  applications 

•  equal  opportunities  to  all  associates 

•  availability  of  internal  mentors  for  projects 

•  frequent  location  change  at  Cognizant  Academy 
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Practices  related  to  process: 


•  internal  quality  reviewers  for  large  projects 

•  accessibility  to  project-specific  intranet 

•  judicious  use  of  central  /  group  review 

•  sharing  of  Lotus  Notes  database  with  clients 

•  Resource  Management  Group — each  one  takes  care  of  one  location 

•  Human  Resources — quarterly  reviews  against  targets 

•  harmonization  of  quality  reviewer  audits 

•  sharing  of  risk  management  plan  with  customer 

•  having  an  independent  delivery  coordinator 

•  case  study  approach  to  training 

•  checklist  for  joint  requirements  planning 

•  documents  organized  stage-wise,  as  in  Project  Software  Process  Handbook,  enabling 
quality  reviewers  to  do  audits  from  their  desktops 

•  having  assessment  centers  away  from  office 

•  adopting  use  case  point  technique  for  effective  estimation 

•  close-to-real  estimation  using  Extended-FP 

Practices  related  to  technology: 

•  effective  use  of  knowledge  repository 

•  accessibility  to  component  repository 

•  availability  of  SQA  ROBOT,  Winrunner  -  testing  and  eraser  tool  customized  from  Y2K 
tool 

•  development  and  use  of  referral  management  tool 

•  use  of  standards  checking  tool  for  Java 

•  metrics-based  status  reporting  system 

•  Web-based  issue  tracking  system 

•  paperless  office  RAMS  tool  has  been  piloted  and  intranet-based  travel  request  system 

•  eZCreator  for  automating  SPP  and  PSPH  preparation  and  automatic  CV  generator 

•  Rational  Rose  for  analysis  and  design 
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Practices  related  to  business: 


•  existence  of  Business  Vision  Group  for  an  account 

•  pioneering  initiatives  at  CA  giving  social  visibility 

•  positioning  process  as  role  model  to  client 

•  creating  brand  equity  in  campuses 

QSmart  is  a  comprehensive  tool  with  the  complete  Software  Quality  Assurance  function 
automated  through  powerful  built-in  workflow  mechanisms.  Key  features  are  online  review 
and  approval  of  project  activities,  meticulous  audit  planning,  and  reporting  activities  with 
provision  to  display  and  monitor  all  SQA-related  metrics  that  reflect  project  health.  It  main¬ 
tains  a  complete  knowledge  repository  for  project  management  activities  and  graphically  dis¬ 
plays  key  profile  parameters  of  projects. 

4.4.4  People  and  Cultural  Issues 

Cognizant  strongly  believes  that  its  associates  form  the  crux  of  its  competitive  advantage  in 
the  market.  Therefore  there  is  a  very  strong  people-centric  orientation  in  the  organization 
with  emphasis  on  continuous  training  and  competency  building. 

In  a  techno-centric  software  environment,  associates  grow  rapidly  in  terms  of  responsibility 
without  much  time  or  opportunity  to  gain  the  required  skills  needed  to  lead  teams,  manage 
projects,  and  network  with  customers.  This  is  addressed  by  providing  a  strong  support  system 
to  help  associates  enhance  their  behavioral  skills  and  competencies.  Several  innovative  inter¬ 
ventions  such  as  role-transition  workshops,  team-building  programs,  cross-cultural  adaptabil¬ 
ity  workshops,  and  leadership  modules  are  conducted  throughout  the  year  to  address  this 
need.  Intranet  sites  on  various  areas,  including  behavioral  skills,  bring  such  competencies  to 
the  desktops  of  the  associates  and  allow  a  widely  dispersed  workforce  ample  opportunity  to 
acquire  these  competencies.  This,  coupled  with  technical  training  on  all  the  frontier  technol¬ 
ogy  areas,  helps  associates  manage  change  and  transition  smoothly.  Every  associate  under¬ 
goes  a  mandatory  ten  days  of  training  in  a  year. 

Assessment  centers  are  also  conducted  every  year.  These  help  identify  the  competency  levels 
of  individuals  and  provide  a  clear  developmental  plan.  Comprehensive  induction  programs 
(spanning  nine  weeks)  held  on  a  continuous  basis,  combine  a  detailed  business,  technical, 
soft-skill  orientation,  and  on-the-job  induction,  which  every  new  entrant  needs  to  undergo  at 
the  time  of  joining. 

A  special  “fast-track”  program  identifies  the  high  performers,  who  are  then  mentored  and 
groomed  to  help  them  maximize  their  potential  and  grow  into  future  leaders.  Mentoring  also 
exists  at  very  senior  levels,  where  managers  are  in  turn  mentored  and  guided  by  the  senior 
management  team  throughout  the  year. 
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Several  “togetherness”  events  such  as  “Annual  Days”  and  excursions  help  foster  strong 
bonds  between  the  employees  and  the  organization.  Other  forums  such  as  “Open  Houses” 
provide  a  platform  for  associates,  irrespective  of  hierarchy,  to  pose  questions  to  the  manage¬ 
ment. 

Retention  rates  in  Cognizant  are  among  the  highest  in  the  country.  The  attrition  rate  at  middle 
and  senior  management  levels  is  almost  zero.  High  retention  is  the  direct  outcome  of  com¬ 
petitive  compensation  packages,  attractive  employee  benefit  schemes,  participation  in  em¬ 
ployee  stock  option  plans,  and  an  assimilative  work  culture. 

4.4.5  Continuing  Improvement 

The  enhancements  to  the  Quality  Systems  are  through  research,  senior  management  direc¬ 
tives,  analysis  of  internal  audit  findings,  project’s  feedback,  internal  support  services  survey, 
customer  satisfaction  survey,  gap  analysis,  and  effectiveness  of  the  enhancement  measured 
through  metrics. 


4.4.6  Summary 

•  Process  benefits:  processes  continuously  and  systematically  improved,  and  common 
sources  of  problems  understood  and  eliminated 

•  Technology  benefits:  new  technologies  proactively  pursued  and  deployed 

•  People  benefits:  strong  sense  of  teamwork  across  the  organization,  and  all  involved  in 
process  improvement 

•  Measurement  benefits:  data  used  to  evaluate  and  select  process  improvements 

While  these  are  the  benefits  achieved  by  CTS  being  at  maturity  level  5,  the  following  areas 

could  be  looked  into  by  the  SEI: 

1.  The  Software  CMM  model,  and  in  particular  the  CBA IPI,  must  have  a  surveillance  type 
model  /  requirement  to  ensure  continued  compliance  to  the  KPAs  at  the  assessed  level. 

2.  The  SEI  should  strongly  recommend  the  delinking  of  the  marketing  focus  of  the  Soft¬ 
ware  CMM  assessment  and  advocate  on  internal  benefits  to  an  organization  with  respect 
to  process  improvement. 

3.  The  SEI  should  conduct  and  authorize  training  in  India  of  the  courses  that  are  currently 
being  done  in  Pittsburgh,  such  as  the  Software  CMM  Lead  Assessor  Course,  PSP,  etc. 

4.  The  SEI  should  enforce  through  their  qualified  assessors  a  very  stringent  assessment  at 
Level  5.  Unless  companies  adequately  demonstrate,  both  during  assessment  and  on  a 
continuous  basis,  the  KPAs  at  Level  5  such  as  Technology  Change  Management,  Defect 
Prevention,  and  Process  Change  Management,  such  companies  must  not  be  declared  to 
be  at  Level  5.  Often  teams  that  worked  for  these  Level  5  KPAs  are  disbanded  once  the 
assessment  is  through. 

5.  The  Software  CMM  model  may  also  have  to  look  into  the  emerging  trends  in  new  tech¬ 
nology  areas  where  cycle  times  are  constantly  shrinking.  Industry  is  looking  forward  for 
“CMM  for  lightweight  processes,” 
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4.5  Computer  Sciences  Corporation  (CSC),  Aegis 
Program,  Moorestown,  NJ 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor(s) 

Point  of  Contact 
Web  Page 

Size  of  the  Organization 
Typical  Program  Size 

Primary  Application  Domain(s) 


5 

March  2001 

Kathryn  Gallucci  (Lead  Evaluator) 

Wendy  Irion  Talbot,  wirionta@csc.com 

www.csc.com 

-400  software  professionals 

Projects  range  in  size  from  50  to  200  individuals 

Project  size  varies,  but  frequently  ranges  2-3  million  SLOC 

These  are  DoD  real  time  combat  systems  in  the  C4I  and  weapons 
domains.  Specific  product  lines  include:  radar,  weapons  control, 
command  and  control,  and  interactive  displays. 


This  program,  with  many  upgrades  and  follow-on  increments,  has  been  in  place  since  1969. 
This  legacy  has  resulted  in  an  organizational  environment  that  has  required  strong,  disci¬ 
plined  management  and  software  practices  for  nearly  30  years.  Formal,  model-based  process 
improvement  efforts  were  begun  in  1993,  Level  2  was  achieved  in  1994,  Level  3  in  1996, 
Level  4  in  1998,  and  Level  5  in  2001.  The  organization  has  undergone  CBA  IPIs,  source  se¬ 
lection  SCEs,  various  internal  assessments  and  has  commissioned  SCEs. 


4.5.1  ROI  and  Improvement  Trend  Data 

We  have  viewed  ROI  from  a  variety  of  perspectives.  We  have  invested  a  full-time  core  staff, 
augmented  by  a  part-time  network  that  spans  the  program,  in  our  process  improvement  pro¬ 
gram.  We  monitor  the  resources  applied  to  process  improvement  activities  against  measurable 
changes  in  performance,  as  well  as  the  intangible  value-added  effects  including  employee 
morale.  Investments  are  linked  to  specific  business  goals  which  range  at  a  high  level  from 
client  satisfaction  on  contract  perfonnance,  or  Software  CMM  rating  required  for  a  new  busi¬ 
ness  opportunity,  to  the  project  implementation  activities,  such  as  specific  performance  im¬ 
provement  of  our  code  inspection  process.  The  process  improvement  activities  are  managed 
and  subjected  to  the  same  rigorous  business  management  practices  as  the  product 
development  activities. 

4.5.2  Barriers  to  Achieving  High  Maturity 

Our  initial  barrier  to  pursuing  high  maturity  practices  in  a  formal,  model-based  form  rested  in 
making  the  business  decision  that  there  would  likely  be  return  on  the  investment  if  this  were 
pursued.  After  achieving  a  Level  3  capability  in  1996,  there  was  a  period  of  research  and  re¬ 
flection  to  support  this  decision  process.  We  recognized  that  there  were  many  elements  of  the 
Level  4  and  5  process  areas  in  place  already  as  a  function  of  the  standard  program  processes. 
However,  some  elements  of  the  key  practices  were  not  in  place.  In  particular,  the  advanced 
analytical  techniques  (e.g.,  statistical  process  control)  were  only  used  in  specific  areas  and 
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instances  in  the  program  as  needed,  and  were  not  part  of  a  defined  overall  quantitative  man¬ 
agement  strategy.  We  needed  to  determine  whether  furthering  the  maturity  of  these  practices 
would  be  advantageous  to  the  program.  Management  needed  to  understand  and  buy-in,  real¬ 
izing  that  some  high  maturity  practices  require  significant  investment  in  terms  of  both  re¬ 
sources  and  lead  time  before  results  can  be  observed  (e.g.,  defect  leakage  reduction  over 
time).  In  addition  to  reviewing  other  organizations’  successes  and  reports  on  ROI,  the  market 
was  also  taken  into  consideration.  Not  surprisingly,  as  more  organizations  achieved  the  Level 
3  rating.  Levels  4  and  5  became  competitive  discriminators.  The  decision  to  proceed  was 
made  and  recognized  as  a  strategic  initiative,  rather  than  a  short-term  tactical  objective. 

Once  the  decision  was  made  to  continue  to  advance  rather  than  just  maintain  Level  3,  obvi¬ 
ous  barriers  included  process  change  management  issues:  process  modification  investment, 
training,  resistance  to  change,  competing  program  vs.  process  improvement  priorities,  and  a 
lack  of  understanding  of  the  true  value  of  institutionalized  advanced  maturity  processes.  In 
retrospect,  the  cultural  issues  were  probably  the  largest  hurdle,  and  are  addressed  below.  An¬ 
other  barrier  which  we  did  not  recognize  as  such  early  on,  but  became  extremely  obvious  as 
we  tackled  it,  was  truly  understanding  which  components  of  our  development  cycle  were  ap¬ 
propriate  for  quantitative  management  using  statistical  process  control  versus  other  quantita¬ 
tive  methods.  There  was  a  lack  of  readily  available  guidance  on  alternate  quantitative  man¬ 
agement  mechanisms  that  were  both  useful  and  would  satisfy  model  criteria,  other  than  SPC. 

4.5.3  Unique  or  Distinguishing  Practices 

After  having  institutionalized  Level  5  throughout  the  projects  on  the  Aegis  program,  it  was 
evident  that  there  were  unique  activities  in  place  that  distinguished  this  organization  from 
itself  in  earlier  years,  and  other  projects  /  programs  not  yet  at  Level  5. 

Senior  Leadership:  In  the  Level  5  organization,  the  highest  level  of  senior  leadership  exe¬ 
cutes  commitment  by  being  personally  involved  and  holding  the  organization  at  each  level 
responsible  for  performing  in  an  advanced  maturity  manner.  The  implementation  of  “com¬ 
mitment”  changes  from  only  ensuring  sufficient  budget  and  resources,  to  also  being  involved 
on  a  personal  level.  They  “walk  the  walk”  and  apply  advanced  maturity  practices  in  their 
daily  activities.  In  addition,  process  improvement  and  project  goals  were  de-conflicted;  these 
goals  were  aligned  in  a  way  that  made  business  sense  for  the  project  and  supported  the  proc¬ 
ess  improvement  objectives. 

Process  Management  Approach:  Process  management  activities  were  distributed  and  sup¬ 
ported  by  a  larger  segment  of  the  population.  While  the  SEPG  remained  the  focal  point  for 
process  improvement,  a  wider  “network”  of  working  groups  and  committees  responsible  for 
specific  processes  not  only  directly  involved  more  people  in  the  process  change  and  man¬ 
agement  activities  but  also  helped  to  ease  the  cultural  issues  and  aided  further  institutionaliza¬ 
tion. 
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Listen  to  the  People:  Today  during  meetings,  regardless  of  subject  and  participants,  we  hear 
talk  about  needing  even  more  and  better  documented  processes,  should  the  Defect  Prevention 
Committee  be  represented  in  the  discussion,  what  type  of  analysis  is  appropriate,  are  the  right 
data  elements  being  analyzed  for  statistical  relationships,  etc.  The  level,  depth  and  content  of 
daily  activities  with  respect  to  advanced  maturity  concepts  has  changed.  “Statistical  thinking” 
is  common  rather  than  the  exception. 

4.5.4  People  and  Cultural  Issues 

We  found  that  we  really  needed  to  address  moving  to  Level  5  in  a  holistic  fashion,  that  is  en¬ 
suring  that  the  entire  process  infrastructure  was  considered  and  developed  as  an  integrated 
whole  and  that  the  staff  understood  the  context  of  the  overall  efforts  in  relationship  to  their 
individual  activities.  As  noted  earlier,  the  cultural  issues  were  probably  the  largest  barrier  to 
advancing  our  maturity.  It  was  rapidly  evident  that  moving  to  Level  5  was  not  just  a  matter  of 
implementing  the  mechanics  of  process  change  -  the  cultural  implications  needed  to  be  iden¬ 
tified,  planned  for,  and  addressed.  Appropriate  training,  updated  formal  mentoring,  regular 
focus  group  discussions  and  an  improved,  self-sustaining  process  infrastructure  facilitated  the 
culture  changes  required. 

In  operating  at  Level  5,  we’ve  found  that  a  broader  segment  of  the  population  becomes  in¬ 
volved  in  process  (and  technology)  improvement,  from  suggesting  change  to  participating  in 
various  initiatives.  There  is  now  a  stronger  belief  that  the  individual  can  initiate  effective 
change. 

The  single  cultural  factor  that  most  directly  affected  our  path  to  success  was  the  sustained 
personal  involvement  of  our  program  manager  (PM).  In  addition  to  supplying  resources  and 
funding  as  needed,  the  PM  participated  in  training  sessions,  focus  group  discussions,  and  ad¬ 
justed  his  management  techniques  and  formal  review  content  to  include  the  advanced  process 
maturity  initiatives.  This  not  only  signaled  that  he  was  fully  committed  to  achieving  the  Level 
5  goal,  but  that  this  wasn’t  just  a  mark  in  the  sand  -  the  practices  were  applied  and  performed 
at  the  highest  levels  of  the  organization  on  a  daily  basis.  This  enlightened  leadership  was  and 
remains  visible  to  the  entire  organization. 

4.5.5  Continuing  Improvement 

Our  efforts  remain  focused  on  continuing  to  improve  performance  in  all  aspects  of  our  pro¬ 
gram  in  a  cost-effective  manner  -  essentially  to  “do  more  for  less.”  Our  process  improvement 
program  applies  not  only  to  the  mainstream  software  development  activities,  but  also  to  the 
infrastructure  and  functional  support  areas  such  as  human  resource  management,  finance  and 
administration,  etc.  Some  specific  initiatives  include: 

•  investigating  new  tools  to  support  qualitative  and  quantitative  analysis 

•  identifying  and  testing  other  areas  in  the  development  cycle  and  in  the  infrastructure  pro¬ 
cesses  where  quantitative  management  may  be  beneficial 
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•  strengthening  and  extending  our  quantitative  analysis  skills,  knowledge  and  resources 

•  enhancing  defect  prevention,  and  particularly  reducing  defect  leakage,  in  both  the  devel¬ 
opment  cycle  and  our  business  management  strategy 

•  assessing  the  impact  of  transitioning  to  CMMI 

•  migrating  our  process  management  approach  to  an  integrated  knowledge  management 
environment 

In  addition,  now  that  we  have  impacted  our  business  culture,  we  are  challenged  to  strengthen 
and  reinforce  our  holistic  “Level  5”  view  so  that  it  withstands  and  continues  to  support  our 
increasingly  dynamic  business  environment. 

The  bottom-line  remains  the  positive  return  on  investment.  Initiatives  must  be  tied  to  clearly 
defined  business  objectives  to  which  measurable  value  can  be  assigned.  We  must  remain  vigi¬ 
lant  and  resist  the  common  practice  of  jeopardizing  long-term  growth  for  the  sake  of  short¬ 
term  profit.  Developing  and  deploying  continuing  improvement  initiatives  is  a  front-end 
loaded  expense,  both  in  dollars  and  human  resources.  Management  must  remain  steady  and 
supportive,  and  staff  must  be  motivated  to  participate  and  develop  a  heightened  awareness  of 
the  strategic  benefit  of  process  improvement  to  the  organization,  to  the  roles  they  perform, 
and  to  themselves  as  individuals. 

4.5.6  Summary 

Now  that  we’ve  moved  to  Level  5,  it  is  increasingly  our  belief  that  the  advanced  maturity 
practices  are  actually  essential  to  effective,  enlightened  and  ultimately  successful  project 
management.  This  is  particularly  true  in  today’s  market  where  rapid  technology  change  is  the 
norm,  innovative  process  and  technology  strategies  are  the  requirement,  and  management  of 
those  strategies  will  determine  survival  and  success. 
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4.6  Computer  Sciences  Corporation  (CSC),  Civil 
Group,  Greenbelt,  MD 


Maturity  Level 

Date  of  Assessment 

Lead  Assessor(s) 

Point  of  Contact 
Web  Page 

Size  of  the  Organization 
Typical  Program  Size 

Primary  Application  Domain(s) 


4  (Civil  Group) 

5  (SEAS  Program  within  Civil  Group) 

Civil  Group — January  2001 

SEAS  Center — ^November  1998 
Paul  Byrnes 

Mel  Wahlberg,  mwahlber@csc.com 
www.cscxom 

Civil  Group  includes  approximately  2100  software  professionals 

Size  varies  so  widely  that  it  is  almost  impossible  to  define  “typical.” 
Individual  projects  range  from  less  than  one  person  up  to  about  200. 

Again  this  information  varies  widely  and  includes  probably  most 
major  architectures  and  systems.  Major  application  domains  include 
air  traffic  control,  NASA  spacecraft  control  systems,  and  tax  systems 
modernization. 


Civil  Group  was  formed  in  June  1998  from  a  reorganization  of  CSC’s  former  Systems  Group. 
Civil  Group  includes  all  of  CSC’s  Federal  Government  business  with  Civil  agencies.  Civil 
Group  was  independently  assessed  at  Level  3  in  May  1999  and  at  Level  4  in  January  2001. 
The  SEAS  organization  within  Civil  Group  was  assessed  at  Level  5  in  November  1998  and  is 
also  ISO  9001  registered. 


As  we  are  maturing  as  an  organization,  we  believe,  increasingly,  that  “high  maturity”  prac¬ 
tices — especially  those  involving  measurement — are  actually  basic  to  effective  management. 
Without  these  types  of  metrics,  an  organization  cannot  assess  the  actual  quality  of  the  prod¬ 
ucts  it  produces  (software  or  otherwise),  or  determine  how  well  its  management  processes  are 
working.  Ultimately,  these  high  maturity  practices  also  become  a  rallying  point  for  personnel 
and  a  motivational  force  for  measuring  the  organization’s  performance. 
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4.7  IBM  Global  Services  India,  Bangalore,  India 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor(s) 

Point  of  Contact 
Web  Page 

Size  of  the  Organization 
Typical  Program  Size 

Primary  Application  Domain(s) 


5 

November  99 
Richard  Storch 

Dr  Asha  Goyal,  gasha@in.ibm.com 
www.ibm.com/in 

2400  software  professionals  (full-time  employees) 

Any  number  from  5  to  30  people  per  typical  project 
300-400  Function  Points  or  parts  of  very  large  products 
Systems  software,  firmware  and  chip  design 
Commercial  software  products 
Information  systems  software  for  business  processes 
Contract  or  outsourced  software  development 
End  user  software,  such  as  Office  Suite 


IBM  Global  Services  India  -  Software  Export  Group  has  been  ISO  9001  certified  since  1994 
when  it  was  a  small  group.  At  that  time  it  was  more  focused  on  system  software  and  operat¬ 
ing  systems.  Since  then  it  has  grown  much  larger,  has  been  re-certified  with  multi-location 
extension  and  is  moving  to  ISO  9001:2000  now.  The  organization  started  a  PSP  initiative  us¬ 
ing  internal  faculty  and  has  50  PSP-trained  people  currently.  PSP  is  now  being  introduced  in 
basic  training.  The  organization  was  assessed  at  maturity  level  4  in  November  1997  and  at 
maturity  level  5  in  November  1999  by  SEI  authorized  Lead  Assessors. 

The  organization’s  primary  business  objective  is  to  meet  commitments  made  to  the  customer. 
This  implies  that  better  predictability,  adherence  to  service  level  agreements,  and  defined 
processes  are  important.  The  organization-wide  baselines  have  been  defined,  and  every  pro¬ 
ject  has  to  meet  them.  The  baselines  are  enhanced  on  the  basis  of  trends.  Individual  projects 
can  take  specific  targets  to  improve.  ROI  currently  has  been  observed  at  4  to  5  times  the  in¬ 
vestment  in  SPI  activities. 


4.7.1  Barriers  to  Achieving  High  Maturity 

Customer  satisfaction  is  very  important.  The  customers  appreciate  the  maturity  practices. 
Customers  do  look  for  specific  measures  and  alignment  in  technology.  This  is  not  exactly  a 
barrier  to  maturity,  but  requires  continuous  pursuit  in  the  right  direction.  Processes  and 
documentation  are  sometimes  perceived  as  overhead,  and  the  customer  may  look  for  immedi¬ 
ate  return  and  may  not  be  satisfied  with  a  long-term  view  in  the  early  part  of  the  project.  This 
barrier  is  crossed  with  the  right  culture  and  senior  management  commitment. 

As  a  culture  in  IT  industry,  the  creative  activity  of  code  writing  is  associated  more  with  ana¬ 
lytical  and  less  with  numerical,  whereas  project  management  uses  more  quantitative  analysis. 
Team  members  at  junior  levels  are  more  intuitive  and  less  oriented  towards  measurements. 
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When  high  maturity  culture  prevails,  a  change  happens  to  this  thinking  and  a  balance  in 
quantitative  and  analytical  can  be  seen.  Practitioners  also  have  to  change  their  role  as  they 
move  from  activities  of  prototype  to  development  and  to  support.  As  this  happens,  the  atti¬ 
tudes  have  to  change,  and  awareness  for  appropriate  measures  is  to  be  created.  There  are  al¬ 
ways  a  lot  of  change  elements  in  the  environment.  All  these  cannot  be  addressed  by  auto¬ 
mated  systems.  These  have  to  be  addressed  as  part  of  process  maturity. 

In  the  published  literature,  very  clear  information  on  investment  and  corresponding  activities 
related  to  SPI  as  well  as  ROI  is  not  available  at  the  desired  granularity.  This  makes  it  difficult 
to  justify  investments  in  business  environment  other  than  as  culture  building.  On  the  subject 
of  ROI,  we  are  working  on  a  special  model  so  that  all  practitioners  can  have  a  mind  share  in 
any  ROI  framework. 

4.7.2  Unique  or  Distinguishing  Practices 

We  have  an  automated  process  change  management  tool  available  to  all  practitioners.  The 
change  requests  are  systematically  analyzed,  prioritized,  and  processed.  The  quality  group 
very  often  pilots  concepts  in  training,  tools  and  processes,  makes  recommendations  for  or¬ 
ganizational  changes,  and  moves  on  to  other  activities.  The  audit  reporting  process  includes 
reporting  on  best  practices  in  different  groups  and  gives  quantitative  performance  so  that 
groups  can  benchmark  with  each  other. 

Every  project  accesses  a  database  containing  learnings  from  various  projects  before  starting 
each  phase  to  avoid  recurrences  of  similar  defects.  Defect  prevention  activities  are  taken  up 
using  effective  statistical  tools  to  analyze  data,  arrive  at  root  causes,  and  measures  to  be  im¬ 
plemented  to  eliminate  these  root  causes  and  minimize  their  occurrence.  There  is  an  exten¬ 
sive  process  assets  library  (tools,  reusable  artifacts,  and  learnings)  that  makes  information 
available  on  a  shared  server.  There  is  extensive  focus  on  training,  resources,  and  tracking. 

An  integrated  measurement  framework  covering  all  relevant  levels  in  the  organization  (senior 
management,  support  functions,  and  projects)  has  been  created.  Quality  policy  is  related  to 
operational  measures  at  project  and  department  level.  Capability  baselines  for  in-process 
quality  are  available  to  practitioners.  A  measurement  program  is  supported  by  close  interac¬ 
tion  with  academicians  and  enables  practitioners  to  appreciate  the  “concepts”  and  their  “im¬ 
plementation”  in  industry.  A  dedicated  group  for  measurement  provides  guidance  and  support 
to  practitioners. 

4.7.3  People  and  Cultural  Issues 

A  useful  cultural  aspect  is  that  of  team  building  and  close  working  of  the  quality  group  with 
practitioners.  They  work  together  and  help  recognize  innovations  that  can  come  up  as  process 
changes.  One  issue  that  comes  up  is  that  of  regular  measurements.  Practitioners  often  make 
the  measurements  but  may  not  submit  them.  Very  generalized  automated  systems  for  data 
collection  sometimes  only  give  gross  measurements.  In  depth  specialized  measurement 
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analysis  studies  with  the  help  of  QA  managers  or  SQAs  are  done  to  bring  out  many  aspects  of 
data.  The  change  management  is  handled  with  the  help  of  groupware  and  by  communicating 
with  practitioners. 

Very  often  employee  turnover  is  the  only  resource  change  aspect  that  gets  attention.  How¬ 
ever,  what  also  needs  attention  is  the  people  movement  upwards,  laterally  and  to  different 
locations  as  they  grow  and  as  business  grows.  We  have  taken  some  specific  measures  to  keep 
people  involved  in  process  changes  and  to  keep  them  aware  of  changes  as  far  as  the  quality 
management  system  is  concerned.  These  measures  also  help  in  process-related  skill  building. 
In  our  formal  and  extensive  training  programs  for  new  hires,  we  have  introduced  quality- 
related  topics  for  awareness.  We  have  a  formal  mentoring  program.  But  mentoring  also  works 
better  with  clear  objectives  and  confidentiality.  Employee  morale  definitely  improves  as  a 
result  of  the  process  improvement  activities  and  is  associated  with  active  participation  of 
practitioners,  middle  and  senior  management  in  process  improvement  activities  (internal  as¬ 
sessments,  workshops,  industry  interactions  through  paper  submissions,  presentations)  apart 
from  just  the  quality  group. 

A  good  practice  called  “Buddy  System”  is  in  practice,  in  which  each  new  joinee  is  associated 
with  a  senior  person  from  the  same  team.  Buddy  prepares  a  plan  for  induction  and  makes  sure 
that  the  joinee  goes  through  full  cycle  of  induction  and  covers  all  related  departments  activi¬ 
ties.  It  is  a  clearly  defined  process  and  is  not  just  an  open  concept.  It  also  defines  how  many 
hours  we  expect  buddy  to  spend  and  how  long  the  relationship  is  expected  to  be  active.  Mo¬ 
rale  of  the  employee  has  increased  considerably  because  of  the  structured  processes  as  well 
as  well  defined  roles  and  responsibilities. 

4.7.4  Continuing  Improvement 

We  have  been  working  extensively  with  Software  CMM  model.  We  do  see  that  it  helps  the 
group  in  a  techno-cultural  manner  to  move  ahead  with  one  model  (like  one  shared  language 
for  practitioners),  a  well-defined  goal,  and  “something  new”  to  achieve.  We  do  intend  to  ex¬ 
plore  CMMI  and  plan  an  assessment  in  early  2002.  We  are  also  continuing  our  process  model 
integration  with  ISO  and  going  for  ISO  9001:2000  certification.  We  have  started  our  initia¬ 
tives  on  People  CMM.  The  biggest  barrier  to  maturity  is  the  flip  side  of  growth  itself,  that  is, 
keeping  an  ever-growing  manpower  resource  operating  with  mature  processes.  It  is  not  only 
time-to-market  that  needs  to  be  reduced  but  also  the  time-to-understand-process  and  become 
competent  in  using  processes  and  process  models.  The  main  improvement  objective  is  to  en¬ 
hance  processes  to  work  with  organizational  initiatives  of  competency  development,  training 
and  technology  induction  while  incorporating  SDLC  innovations  to  address  business  scenar¬ 
ios. 

4.7.5  Summary 

Market  forces  make  an  organization  aware  of  where  it  stands  and  it  shows  up  in  business 
growth.  What  the  organization  has  to  do  itself  is  to  find  out  how  to  improve  and  then  make 
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the  improvement  happen.  We  have  observed  that  being  at  maturity  level  4  fosters  a  culture  of 
measurement  appropriate  to  the  problem  at  hand  and  hence  close  to  a  practical  usable  level.  It 
helps  the  management  and  those  who  are  not  involved  at  a  certain  level  of  technical  detail,  to 
see  the  trends  in  delivery.  Then  the  management  can  get  a  feel  for  what  is  happening  and  lead 
the  directions  in  an  overall  manner.  This  also  allows  everyone  to  understand  how  different 
parts  of  the  organization  are  doing  and  enables  a  culture  of  improvement.  If  the  measurement 
framework  has  the  right  granularity  then  one  can  see  where  the  roadblocks  are.  The  same 
people  who  are  facing  the  roadblocks  also  create  the  innovations  to  remove  them  because 
they  can  see  the  picture  clearly. 

Being  at  maturity  level  5  helps  one  in  being  proactive  in  the  areas  addressed  by  Process 
Change  Management  (PCM),  DP,  and  TCM.  PCM  helps  in  understanding  where  the  change 
is  coming  from.  Then  the  change  management  can  be  looked  at  in  a  holistic  manner  and  or¬ 
ganizational  action  can  be  taken.  It  can  almost  help  giving  a  direction  for  organization  devel¬ 
opment  activities.  DP  is  by  far  the  most  important  activity  due  to  changes  that  are  happening 
at  any  point  of  time  and  how  one  can  hardly  cope  with  them.  Probably,  more  academic  work 
needs  to  be  done  in  this  area.  One  can  analyze  the  problems,  but  are  we  able  to  reduce  the 
repetition  of  those  problems?  Are  we  able  to  use  technology  to  solve  those  problems?  Can  we 
create  measures  that  help  us  make  decisions  to  solve  those  problems?  It  does  help  in  making 
people  aware  of  the  forthcoming  problems  and  also  makes  them  confident  that  something  can 
be  done. 

TCM  has  been  well  appreciated  by  management  because  it  is  one  of  the  highest  concern  areas 
and  involves  very  visible  costs.  It  gives  the  middle  managers  a  way  to  mold  the  organiza¬ 
tional  thinking,  process,  and  planning  to  move  in  the  desired  direction. 
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4.8  i-flex  solutions  limited,  Mumbai  and  Bangalore, 
India 


Maturity  Level 
Date  of  Assessment 


Lead  Assessor(s) 


Point  of  Contact 
Web  Page 

Size  of  the  Organization 
Typical  Program  Size 

Primary  Application  Domain(s) 


5 

Dec  1995 — CITIL,  Mumbai,  Level  4 
Dec  1995 — CITIL,  Bangalore,  Level  4 

Oct  1999 — CITIL  Data  Warehouse  Center  of  Excellence,  Bangalore, 
Level  5 

Dec  2000 — IT  Services  Division,  Mumbai,  Level  5 

Dec  2000 — -IT  Services  Division,  Bangalore,  Level  5 

Dec  1995 — Cynthia  Wise  and  Ken  Dymond 

Oct  1999 — Ken  Dymond,  S  Santhanakrishnan,  Anand  Kumar 

Dec  2000 — S  Santhanakrishnan,  Atul  Gupta,  Anand  Kumar 

Anand  Kumar,  Anand.Kumar@iflexsolutions.com 

www.iflexsolutions.com 

1400 

10-40  people  per  project 
50-1000  KLOC  per  project 

Banking  products  and  services — retail,  corporate,  investment,  bro¬ 
kerage,  data  warehouse,  Internet 


i-flex  solutions  limited  was  earlier  known  as  Citicorp  Information  Technology  Industries 
Limited  (CITIL).  The  organization  was  assessed  at  Level  4  through  two  back-to-back  as¬ 
sessments  carried  out  in  December  1995,  one  for  each  of  its  sites,  Mumbai  and  Bangalore. 
The  CITIL  Data  Warehouse  Center  of  Excellence,  Bangalore  was  assessed  at  Level  5  in  De¬ 
cember  1999.  The  IT  Services  Divisions  at  Mumbai  and  Bangalore  were  assessed  at  Level  5, 
once  again  through  two  back-to-back  assessments,  in  December  2000. 


4.8.1  ROI  and  Improvement  Trend  Data 

The  following  are  some  ROI  trends  that  the  organization  has  been  able  to  establish  over  the 
years  that  it  has  implemented  a  metrics  program: 

•  mean  annual  growth  in  productivity  of  5%  over  the  last  6  years 

•  mean  annual  reduction  in  defect  density  of  15%  over  the  last  6  years 

•  percentage  rework  effort  halved  over  the  last  5  years 

•  price  of  non-conformance  down  by  50%  over  the  last  5  years 

•  ROI  of  100%  in  the  first  year;  now  stable  at  about  300% 
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4.8.2  Barriers  to  Achieving  High  Maturity 

The  following  are  some  areas  that  the  organization  had  to  grapple  with,  in  its  journey  on  the 
path  of  the  Software  CMM: 

•  non-repetitive  nature  and  scope  of  work  coming  from  hundreds  of  customers.  This  is  an 
outcome  of  being  a  products  company  in  the  area  of  banking  and  financial  software, 
where  the  market  is  very  aggressive. 

•  high  growth  rate,  in  terms  of  revenue,  geographical  dispersion  of  operations,  as  well  as 
manpower. 

•  a  large  inflow  of  new  practitioners,  who  need  to  be  quickly  assimilated  into  the  organiza¬ 
tion’s  process  culture. 

•  reducing  life  spans  for  methodologies  and  technologies. 

•  business  pressures  to  improve  further  even  before  the  full  benefits  of  current  process  im¬ 
provement  initiatives  have  accrued. 

4.8.3  Unique  or  Distinguishing  Practices 

Some  practices  that  we  have,  which  we  believe  would  be  common  to  several  high  maturity 
organizations: 

•  managing  continuous  inflow  of  requirements,  without  adversely  impacting  delivery  ca¬ 
pability 

•  release  cycles  as  low  as  4-6  weeks 

•  evolutionary  life  cycles  piloted,  evaluated,  and  found  effective  in  meeting  time-to-market 
requirements 

•  centralized-cum-dispersed  SEPG  responsibilities 

•  planned  reuse  of  design  and  code  components  across  the  organization 

•  software  rating  using  an  empirical  model 

•  intranet  as  the  primary  communication  vehicle  for  process  improvement  activities  and 
results  e.g.,  process  changes,  process  pilots.  Process  Change  Control  Board 

•  high  level  of  automation  of  software  engineering  and  management  activities 

•  end-to-end  automation  for  data  capture 

4.8.4  People  and  Cultural  Issues 

An  important  challenge  is  to  maintain  and  improve  the  process  culture  of  the  organization 
despite  the  rapid  manpower  induction. 

There  is  a  well-established  and  focused  induction  training  program  for  all  new  joinees,  lateral 
and  freshers. 
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Process  adherence  needs  to  be  constantly  emphasized  through  suitable  rewards  and  punish¬ 
ments  linked  to  this  area. 

Participation  in  process  initiatives  and  programs  is  valued  by  the  organization. 

We  have  a  rotation  policy  that  facilitates  movement  of  people  between  SQA  and  SEPG  on  the 
one  side  and  delivery  on  the  other. 

A  clear  organizational  policy  that  makes  a  term  in  SEPG  or  SQA  mandatory  before  promo¬ 
tions  to  senior  management  cadre  can  be  considered. 

4.8.5  Continuing  Improvement 

The  Software  CMM  will  continue  to  guide  us  in  our  process  framework.  We  now  plan  to  also 
focus  on  knowledge  management  as  an  organizational  initiative.  We  have  been  working  on 
this  for  the  last  18  months,  and  have  already  seen  some  very  positive  results.  In  addition,  we 
are  currently  in  the  process  of  holding  discussions  with  senior  management  on  the  next  quan¬ 
tum  jump  in  our  process  and  quality  initiatives.  Some  options  being  discussed  are:  Six  Sigma, 
PSP  and  TSP,  People  CMM,  CMMI,  and  Balanced  Scorecard. 

4.8.6  Summary 

What  does  it  mean  to  be  maturity  level  4  or  5?  Well,  to  start  with  it  means  that  the  SEI  classi¬ 
fies  you  under  the  umbrella  of  “High  Maturity  Organizations.”  But  what  it  really  means  is 
that  you  start  seeing,  and  therefore  believing  in,  the  real  benefits  of  implementing  the  Soft¬ 
ware  CMM.  Levels  2  and  3  of  the  Software  CMM  can  be  considered  to  be  the  sowing  and 
growing  phases  of  cultivation,  where  you  do  all  the  hard  work  and  lay  the  foundation  for  a 
great  harvest.  Levels  4  and  5  are  where  you  really  reap  the  benefits  of  all  that  hard  work,  in 
quantitative  terms.  You  know  your  process  capability,  and  you  are  able  to  use  the  DP,  TCM, 
and  PCM  infrastructure  to  make  significant  enhancements  to  that  capability  in  order  to  meet 
the  needs  of  the  marketplace.  In  fact,  while  the  Software  CMM  stresses  the  need  to  align 
process  improvement  goals  to  business  goals  across  levels,  the  alignment  becomes  so  natural 
at  Level  5.  From  the  SEPG  point  of  view,  “selling”  processes  to  senior  management  and  pro¬ 
ject  teams  in  a  Level  5  organization  is  so  much  easier  than  doing  the  same  in  a  Level  2  or¬ 
ganization,  for  example. 


40 


CMU/SEI-2001-SR-014 


4.9  Litton/PRC,  McLean,  VA 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor(s) 

Point  of  Contact 
Web  Page 

Size  of  the  Organization 
Typical  Program  Size 


Primary  Application  Domain(s) 


5 

March  2000 
Joseph  Morin 

A1  Pflugrad,  pflugrad_al@prc.com 
www.prc.com 

5500  employees,  2500  are  within  the  scope  of  Software  CMM  ef¬ 
forts 

6  people  per  project,  where  the  typical  project  is  a  single-year  or 
annualized  task  order. 

50-200  KSLOC/year 

Litton/PRC  spans  two  major  domains: 

•  software  and  services  for  National  Defense  Systems 

•  software  and  services  for  Civil  Government  Systems 


In  March  2000,  PRC  received  a  PRC-wide  Level  5  rating  in  which  the  assessment  team  rated 
at  the  practice  level  for  all  process  areas  in  the  Software  CMM  vl.l  model.  This  major  mile¬ 
stone  is  the  last  of  many.  PRC  initiated  model-based  process  improvement  in  1993.  PRC  sites 
attained  Level  2  in  December  1995,  and  PRC  sectors  achieved  Level  3  in  June  1996.  PRC 
developed  a  combined  SE/SW  model-based  program  in  June  1997  and  secured  its  initial  ISO 
9001/9002  registrations  in  May  1998.  In  addition  to  other  ISO  registrations,  PRC  received  a 
PRC-wide  Level  3  rating  in  June  1999. 


PRC  has  integrated  many  different  quality  approaches  into  one  multi-faceted  quality  infra¬ 
structure:  CMM-based  process  improvement,  quality  improvement  (Qualtec  TQM),  ISO 
9000,  quality  assurance,  customer  satisfaction,  and  employee  satisfaction.  The  quality  im¬ 
provement  facet  contains  the  foundational  principles,  teams,  and  methods  upon  which  all 
other  facets  are  built.  PRC  has  pioneered  the  integration  of  the  Systems  Engineering  CMM, 
SECM,  and  Software  CMM  and  has  participated  in  the  development  of  the  CMMI  frame¬ 
work  and  associated  models  and  representations. 


4.9.1  ROI  and  Improvement  Trend  Data 

PRC’s  budget  for  engineering  process  improvement  has  exceeded  $1M  per  year  since  1993. 
This  figure  is  supplemented  by  various  line  expenditures.  The  following  characterize  the 
business  benefit  PRC  has  received  since  its  first  process  capability  baseline  (September 
1999): 

•  defects  in  delivered  documentation  are  down  78% 

•  defects  in  delivered  code  are  down  70% 

•  defects  found  operationally  are  down  60% 

•  PRC’s  ability  to  estimate  costs  on  a  monthly  basis  has  increased  32% 
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•  PRC’s  ability  to  meet  monthly  cost  goals  increased  by  40%  (CPIm  is  0.977;  1.000  is 
where  planned  monthly  costs  equal  actual  monthly  costs) 

•  PRC’s  ability  to  meet  monthly  schedule  goals  increased  by  7%  (SPIm  is  0.980;  1.000  is 
where  planned  monthly  schedule  equals  actual  monthly  schedule) 

4.9.2  Barriers  to  Achieving  High  Maturity 

To  achieve  Level  4,  PRC  had  to  overcome  several  barriers. 

First,  PRC  needed  to  resist  applying  Level  4  only  to  software-related  activities.  Instead,  we 
adopted  the  Level  4  requirements  to  the  broader  business  issues  of  profitability  and  business 
development  based  on  past  and  present  performance.  These  business  issues  transcended 
software  development  and  yet  still  could  be  applied  to  it.  PRC  worked  to  select  the  few  goals 
and  measures  that  were  most  meaningful  to  all  projects  and  that  had  a  sufficient  stream  of 
continuous  data. 

Secondly,  PRC  needed  to  resist  applying  only  organizationally  mandated  goals  and  measures 
on  projects.  Through  pre-assessment  consultation,  PRC  realized  that  quantitative  manage¬ 
ment  should  be  applied  to  a  project’s  “points  of  pain.”  When  projects  discovered  that  quanti¬ 
tative  management  could  address  the  very  real  problems  they  were  facing,  resistance  to  im¬ 
plementation  changed  to  enthusiasm. 

Thirdly,  PRC  needed  to  think  quantitatively  -  to  value  quantitative  management  and  to  see 
applications  of  it  to  existing  problems.  While  this  ability  comes  with  practice,  it  was  difficult 
to  envision  the  end  result  during  initial  implementation  of  Level  4  principles. 

4.9.3  Unique  or  Distinguishing  Practices 

Some  distinguishing  practices  of  high  maturity  organizations  include: 

•  performing  process  improvement  for  business  reasons,  not  just  process  maturity  goals 

•  managing  by  fact 

•  respecting  people 

•  applying  process  improvement  principles  to  non-model  areas 

•  reducing  and  simplifying  processes  and  process  assets  for  widespread  use 

•  leveraging  corporate  infrastructure,  past  improvements,  and  best  practices 

4.9.4  People  and  Cultural  Issues 

On  one  hand,  PRC  has  historically  maintained  that  the  principles  of  quality  organizations  can 
be  applied  regardless  of  the  organizational  process  maturity.  That  is  why  PRC  implements 
respect  for  people,  managing  by  fact,  continuous  improvement,  and  customer  satisfaction  as 
foundational  principles  for  all  projects  and  teams. 
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On  the  other  hand,  as  PRC’s  process  improvement  program  has  matured,  it  has  had  to  main¬ 
tain  momentum  and  move  from  a  program  targeted  to  innovators  to  one  targeted  at  the  major¬ 
ity  of  managers  and  staff.  Process  improvement  personnel  are  now  assigned  to  various  levels 
of  management,  much  as  contract  and  HR  personnel  are. 

4.9.5  Continuing  improvement 

PRC  is  actively  pursuing  improvements  to  increase  project  performance  within  its  major 
business  areas.  First,  PRC  is  implementing  widespread  use  of  quantitative  management 
within  all  organizational  units.  PRC  management  has  begun  rollout  and  review  of  PRC-wide 
quantitative  project  management  initiatives  for  given  types  of  projects  and  values  during 
monthly  operational  reviews.  Second,  PRC  is  adding  processes  and  process  improvement 
support  for  non-model  process  areas  such  as  information  assurance,  COTS  integration,  net¬ 
work  management,  transition  planning,  database  administration,  etc.  PRC  believes  that  the 
CMMI  is  a  process  framework  flexible  enough  to  add  support  for  these  engineering  proc¬ 
esses,  and  therefore,  PRC  is  actively  transitioning  to  full  CMMI  implementation.  Third,  PRC 
is  pre-tailoring  corporate  processes  to  program  units  to  reduce  or  eliminate  the  amount  of  tai¬ 
loring  necessary  at  the  task  order  or  small  project  level  within  our  business  domains.  Finally, 
PRC  is  refining  its  internal  assessment  methods  to  include:  1)  targeted  assessments  using  a 
subset  of  CMMI  process  areas  within  the  continuous  representation,  and  2)  informal  interim 
assessments  based  upon  periodic  Q A  process  audits. 

4.9.6  Summary 

To  adapt  Winston  Churchill,  Level  5  is  not  the  end;  it  is  not  even  the  beginning  of  the  end; 
but  it  may  be  the  end  of  the  beginning.  Level  5  gives  an  organization  the  tools  it  needs  to  in¬ 
dependently  and  continuously  address  and  resolve  its  own  business  issues.  Specifically,  Level 
5  gives  PRC  the  ability  to  manage  by  fact,  to  quantitatively  increase  performance,  and  to  pro¬ 
vide  process  power  to  each  employee. 
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4.10  Lockheed  Martin  Aeronautics  Company,  Fort 
Worth,  TX 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor(s) 
Point  of  Contact 
Web  Page 


Size  of  the  Organization 
Typical  Program  Size 


Primary  Application  Domain(s) 


4 

December  1999 

Leia  Bowers  White 

Phil  Gould  philip.c.gould@lmco.com 

http://sepg.insite.lmtas.lmco.com/home/index.htm  is  inside  the  LM 
firewall 

http://www.lmaeronautics.com/  is  outside  the  firewall,  but  no  proc¬ 
ess  information  is  there 

1200  during  the  assessment,  approximately  2000  now  after  the 
merger 

20-30  people,  approximately  500KSLOC,  however,  our  projects  do 
range  from  very  small  (2  people  and  2  KSLOC  to  100  people  and  1  + 
MSLOC) 

Military  software 

•  avionics 

•  flight  controls 

•  vehicle  management  systems 

•  stores  management  systems 

•  test  stations 

•  mission  planning 


We  started  our  software  process  improvement  activities  in  early  1991.  We  were  then  General 
Dynamics.  We  achieved  Level  3  in  1993,  ISO  9001  certified  in  1996,  and  ISO  TickIT  in 
1997.  We  achieved  our  Level  4  rating  in  December  1999  as  Lockheed  Martin  Tactical  Air¬ 
craft  Systems.  We  are  currently  merging  three  Lockheed  Martin  companies  (see  the  “Barri¬ 
ers”  section  below)  into  one. 
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Figure  7:  LMAero  Software  Process  Roadmap 

4.10.1  ROI  and  Improvement  Trend  Data 


Avionics  Software  Qualify 


0.09 


*  Total  KSLOC  -  Thousand  Source  Lines  of  Code  (original  baseline  +  new  +  modified) 


Figure  8:  LMAero  Avionics  Software  Quality 
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Avionics  Software  Quality 


I  MovAvq  Def  Dtftsitv  UCL  LCL  ~ — ~  Centeihne  Trend! 


Figure  9:  LMAero  Avionics  Software  Quality 


SOFTWARE  DEVELOPMENT  PRODUCmaTY 


1 .  Prime  Devebpment  Labor  =  "touch  labor"  for  software  design,  code  and  software  test 

2.  CLOC  =  Changed  Line  of  Code  (new  or  modified) 


Figure  10:  LMAero  Software  Development  Productivity 
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Avionics  Delivery  Performance 
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Figure  11:  LMAero  Avionics  Delivery  Performance 

The  above  charts  show  the  history  of  the  Lockheed  Martin  Aeronautics  Company,  Fort 
Worth,  software  process  improvement  efforts.  Our  early  efforts  were  strictly  cost  and  sched¬ 
ule  focused,  while  maintaining  the  existing  quality. 

4.10.2  Barriers  to  Achieving  High  Maturity 

Our  most  significant  barriers  to  achieving  and  maintaining  high  maturity  are  the  continued 
education  and  re-education  of  senior  management.  Senior  management  believes  (and  rightly 
so)  that  this  plant  is  a  builder  of  aircraft  (F16s,  etc.)  Software  is  only  a  part  of  the  airplane 
and  thus  must  be  managed  as  such.  Software  is  approximately  25%  of  our  total  business  base. 

We  successfully  achieved  our  Level  4  assessment  in  December  1999,  after  a  struggle  of  three 
concentrated  years  of  effort.  When  we  started  our  Level  4  effort  (it  was  customer  driven),  our 
internal  assessment  was  that  we  had  slipped  back  from  our  Level  3  rating.  We  completely  re¬ 
built  our  data  collection  system,  we  had  to  re-educate  a  lot  of  people,  and  we  had  a  significant 
management  hill  to  climb.  We  are  now  getting  a  lot  of  management  pull  for  information  that 
can  only  be  provided  by  the  efforts  put  in  place  for  the  Level  4  assessment. 

The  biggest  barrier  we  now  have  is  the  consolidation  of  three  large  Lockheed  Martin  compa¬ 
nies,  and  the  move  to  a  consolidated,  single  set  of  processes.  Right  at  the  time  of  our  Level  4 
assessment,  we  received  word  that  what  was  Lockheed  Martin  Tactical  Aircraft  Systems  (Fort 
Worth,  TX),  Lockheed  Martin  Aeronautical  Systems  (Marietta,  GA),  and  Lockheed  Martin 
Skunkworks  (Palmdale,  CA)  would  be  merged  into  one  company  with  one  president  and  one 
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set  of  processes.  We  are  a  year  into  this  merge  of  three  different  mindsets,  three  different 
process  sets,  three  different  cost  accounting  systems,  three  different  metrics  systems,  three 
different  maturity  levels,  etc.  One  of  the  requirements  of  the  merger  is  that  one  “site”  would 
not  dominate  the  new  company.  The  best  of  each  site  was  to  be  brought  forward.  The  plan  is 
for  a  formal  assessment  in  late  2002,  probably  using  the  SCAMPI  as  our  assessment  method 
with  the  CMMI  approach  as  the  model. 

4.10.3  Unique  or  Distinguishing  Practices 

We  have  a  single  database  (our  internal  built  AutoMet)  that  is  the  single  repository  of  organi¬ 
zation  data.  It  allows  us  to  pull  and  look  at  data  through  multiple  filters  and  to  “roll-up”  data 
from  a  project  view  through  a  program  view  to  an  organization  view,  along  with  getting  a 
domain  view  across  projects  /  programs. 


Figure  12:  LMAero  Data  Aggregation 

Another  area  is  in  our  defect  predictions.  As  can  be  seen  from  the  charts  in  the  ‘Trends”  sec¬ 
tion,  our  cost,  quality,  and  schedule  performance  are  very  much  under  control  and  repeatable. 
Our  next  step  was  to  be  able  to  predict  quality  as  we  progress  through  the  development  life 
cycle.  Below  are  two  slides  from  the  presentation  we  gave  at  SEPG  2001,  which  show  our 
defect  processing  model  and  the  profile  that  we  developed.  This  profile  is  updated  monthly  as 
the  project  continues  through  the  life  cycle.  We  have  seen  an  ability  to  predict  the  amount  of 
test  (and  in  some  cases  actually  decrease  our  test  time)  by  understanding  how  this  model  and 
the  resultant  profile  affect  a  given  project. 
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Figure  13:  LMAero  Data  Processing  Model 


Figure  14:  LMAero  Defect  Profile  Example 
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4.10.4  People  and  Cultural  Issues 

One  thing  we  learned  during  our  journey  to  Level  4  was  that  management  buy-in  is  abso¬ 
lutely  critical.  Even  through  we  have  a  very  “vocal”  and  motivated  process  improvement 
team,  without  the  management  buy-in,  our  efforts  to  get  projects  to  collect  data  differently, 
more  data,  training  on  SPC,  etc.,  would  not  have  succeeded. 

4.10.5  Continuing  Improvement 

Our  next  (continuing)  effort  is  the  merger  of  the  three  companies.  We  need  to  bring  our  other 
processes  (systems  engineering,  hardware  /  electrical,  etc.)  along  with  the  software  process  to 
give  a  complete  product  development  process  view  to  us,  our  senior  management  and  to  our 
customers.  We  will  be  performing  an  assessment  across  the  three  sites  within  the  next  year  or 
so. 


4.10.6  Summary 

LMAero  is  moving  ahead  on  a  path  of  continuous  process  improvement.  We  are  developing 
and  deploying  an  integrated  product  development  process  that  will  join  together  all  disci¬ 
plines  that  work  towards  actual  product  development.  Our  emphasis  will  be  on  “other”  engi¬ 
neering  disciplines  while  maintaining  our  software  process  maturity,  and  using  our  knowl¬ 
edge  of  process  improvement  in  software  to  “speed  up”  the  process  improvement  in  our  other 
engineering  disciplines. 


4.11  Lockheed  Martin  Management  &  Data  Systems, 
King  of  Prussia,  PA 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor(s) 


Point  of  Contact 
Web  Page 

Size  of  the  Organization 
Typical  Program  Size 


5 

December  2000 
Andy  Felschow 
Carol  Granger-Parker 
Dennis  Ring, 

M.  Lynn  Penn,  Mary, lynn.penn@lmco.com 
http://www.lockheedmartin.com 
4500  organization  software  engineers 
9-1500  engineers  per  project 
lOOM  lines  of  code 


Assessed  at  maturity  level  5  in  December  2000,  at  maturity  level  4  in  December  1998,  at  ma¬ 
turity  level  3  in  August  1996,  at  maturity  level  2  in  May  1995.  ISO  9001  certified  and  as¬ 
sessed  against  the  Systems  Engineering  CMM  at  Level  5,  November  2000.  Total  employees 
at  M&DS  are  8500  and  nine  different  geographic  locations,  not  including  customer  sites.  We 
are  currently  adopting  the  People  CMM. 
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4.11.1  ROI  and  Improvement  Trend  Data 

For  a  number  of  years,  Lockheed  Martin  Management  &  Data  Systems  (LM-M&DS)  has  had 
long-term  goals  for  improving  software  productivity  and  quality.  Productivity  is  tracked  in 
terms  of  lines  of  code  /  hour,  while  quality  is  tracked  in  terms  of  defects  /  KLOC  detected 
during  independent  testing  activities.  The  latter  was  regarded  as  a  predictor  of  defect  density 
in  the  field.  Various  strategies  were  employed  to  address  these  goals  including  increased 
process  discipline  and  use  of  quantitative  management  during  our  Software  CMM  Level  4 
period.  However,  the  emphasis  on  early  defect  detection  via  increased  use  of  work  product 
inspections  and  defect  prevention  driven  by  causal  analysis  during  our  Software  CMM  Level 
5  efforts  seems  to  have  had  the  most  impact.  LM-M&DS  believes  there  is  still  much  addi¬ 
tional  benefit  to  be  obtained  as  we  become  better  at  such  activities. 

The  solid  lines  in  the  following  chart  provide  Rayleigh-curve  based  models  of  our  baseline 
performance  for  defect  detection  over  the  life  cycle  as  well  as  our  recent  performance. 


Prelim  Design  Detailed  Design  Code  &  Unit  Test  CI&V,  Major  CI&V,  System  Deployment  Latent 

Component 


Figure  15:  LM  M&DS  Defect  Detection  Profiles  and  Software  CMM  Levels 


The  dashed  lines  are  target  profiles  derived  from  the  baseline  model.  One  profile  reflects  just 
the  impact  of  improved  early  defect  detection  and  the  other  profile  reflects  early  defect  detec¬ 
tion  and  defect  prevention.  The  use  of  such  target  profiles  enabled  programs  to  set  realistic 
goals  for  defect  detection  during  early  phases  and,  as  they  utilized  wider  use  of  inspections 
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and  performed  defect  detection,  they  grew  more  confident  in  the  techniques  and  more  anx¬ 
ious  to  achieve  the  most  ambitious  profile. 

Our  bottom-line  results  against  our  productivity  and  quality  goals  are  reflected  by  the  charts 
in  Figure  16  and  Figure  17. 


The  mature  Software  CMM  Level  5  column  in  each  chart  reflects  our  future  target  as  we  con¬ 
tinue  to  employ  rigorous  defect  prevention  across  all  our  disciplines.  M&DS  recognizes  that 
even  Level  5  is  a  journey  and  that  maturity  within  the  level  is  a  reality. 

4.1 1 .2  Barriers  to  Achieving  High  Maturity 

The  biggest  challenge  for  M&DS  was  the  acquisitions  made.  We  found  ourselves  acquiring 
different  groups  with  different  product  lines,  different  maturity,  and  different  geographic  lo¬ 
cations.  This  challenge  forced  us  to  re-evaluate  our  mandatory  level  of  compliance.  Where 
we  had  taken  a  standard  process  and  through  years  of  maturing  that  process  had  made  it  quite 
specific,  we  were  required  to  raise  it  back  up  and  allow  for  more  tailoring.  This  level  of  com¬ 
pliance  issue  was  for  processes,  metrics,  and  analysis  methods. 
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CMM  Impact  on  M&DS  Defect  Density  During  Test 


I  I 

CMM  Level  4  CMM  Level  5  Current  CMM  Level  5  Mature 


M  Defect  Density  as  a  %  of  Level  3  Defect  Density 


Figure  17:  Software  CMM  Impact  on  LM  M&DS  Defect  Density  During  Test 

4.1 1 .3  Unique  or  Distinguishing  Practices 

In  Process  Quality  (IPQ)  is  a  way  of  looking  at  quality  throughout  the  whole  process  and 
product  development  not  just  at  defined  points.  As  we  state  in  our  introductory  training  “IPQ 
enhances  M&DS’  ability  to  produce  quality  products  by  emphasizing  a  focus  on  quality 
throughout  all  stages  of  development.  It  enables  a  process  of  defect  analysis  and  corrective 
action  to  be  implemented  earlier  in  the  development  cycle.”  The  four  basic  elements  of  IPQ 
are 

•  Discipline  -  defects  do  not  pass  through  to  next  phase 

•  Quality  Awareness  -  quality  goal  setting  in  each  process  phase 

•  Quality  Inspection  -  inspection  of  each  product  and  defect  in  each  process  phase 

•  Root  Cause  Analysis  and  Corrective  Actions 

This  process,  its  definition,  discipline,  and  execution,  were  critical  to  our  achievement  of 
Level  5.  IPQ  has  been  responsible  for  a  major  shift  in  our  Defect  Detection  Profile  to  a  much 
earlier  detection  rate.  Defects  have  dropped  in  test  by  30%. 

4.1 1 .4  People  and  Cultural  Issues 

Similar  to  our  challenge  above  based  on  the  diversity  in  regions  and  product  lines,  people  and 
cultures  were  also  very  different.  Once  the  alignment  of  the  organization  came  together  we 
could  proceed  with  maturing.  We  have  adopted  the  People  CMM  to  put  more  structure  in  our 
workforce  practices.  We  have  formally  adopted  structured  Integrated  Product  Development 
Teams  on  most  large  projects.  We  have  a  formal  mentoring  program  and  have  structured  our 
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core  competencies  to  both  our  strategic  plan  and  our  technology  plan.  We  have  adopted  a 
skills  database  that  is  product  and  function  oriented.  Orientation  training  has  remained  a  very 
focused  formal  process,  which  it  has  always  been.  Our  process  orientation  training  is  now  a 
mandatory  compliance  module  every  year  for  all  employees — discussing  the  latest  in  process 
improvements  at  M&DS.  Employees  are  excited  about  the  process  environment  and  actually 
are  very  responsive  now  to  customer  evaluations  to  show  their  “stuff.” 

4.11.5  Continuing  Improvement 

There  are  numerous  initiatives  at  the  current  time.  We  have  put  together  a  formal  plan  on 
maintaining  Systems  Engineering  CMM  and  Software  CMM  Level  5s  since  we  know  sliding 
back  is  always  very  easy.  We  have  a  formal  one-week  activity  that  we  do  in  each  region  once 
a  year  to  make  sure  we  are  all  still  operating  as  expected — benchmarking  against  the  CMMs. 
We  have  also  integrated  CMM  compliance  into  our  ISO  internal  audit  program.  We  are 
adopting  the  People  CMM.  Also  we  are  looking  at  transitioning  formally  to  the  CMMI  in 
2002  and  the  new  ISO  9001:2000  standard  in  2003. 


4.11.6  Summary 

The  true  benefit  is  in  the  setting  of  expectations.  We  know  what  we  can  do.  We  know  our 
process  capability  as  it  relates  to  today  and  we  can  show  management  how  a  new  process  will 
relate  to  tomorrow.  We  prevent  programs  from  turning  ‘Ted”  — going  bad  versus  reacting  af¬ 
ter  the  fact.  We  know  where  we  are  and  how  we  are  doing,  based  on  data  not  judgment. 


4.12  Lockheed  Martin  Naval  Electronics  and 
Surveillance  Systems-Undersea  Systems 
(Manassas,  VA) 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor(s) 


Point  of  Contact 
Web  Page 

Size  of  the  Organization 
Typical  Program  Size 


Primary  Application  Domain(s) 


5 

February,  1999 
Judah  Mogilensky 
John  Travalent 
Donald  White 

Dana  Roper,  dana.roper@lmco.com 
http://www.lockheedmartin.com/manassas/ 

Approximately  190  software  engineers 

Average  number  of  software  engineers  per  program  =  7 

Average  number  of  delivered  source  statements  per  program  =  1.2 
million 

•  Military  software,  adhering  to  DoD  or  MoD  standards 

•  Systems  software,  used  to  control  physical  devices 
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Lockheed  Martin  Naval  Electronics  &  Surveillance  Systems  -  Undersea  Systems  (LM 
NE&SS-USS)  Manassas  achieved  the  following  certifications: 

1 .  ISO  9001 : 1994  in  September,  1995 

2.  AS  9000: 1997  in  November,  1997 

3.  ISO  14001 : 1996  in  December,  1998 

4.  ISO  9001 :2000  in  December,  2000 

LM  NE&SS-USS  Manassas  has  been  involved  in  process  assessments  for  quite  some  time,  as 
illustrated  in  Table  1. 


Table  1:  LM  NE&SS-USS  Assessment  History 


Date 

Activity 

Results 

£valuator(s) 

September  1990 

Process  Assessment  (SPA) 

Level  3 

Internal;  multiple  divisions 
plus  SEI  personnel 

September  1990 

Capability  Evaluation  (SCE) 

>  Level  2;  Awarded 
contract 

US  Navy 

May  1992 

Capability  Evaluation  (SCE) 

Unknown;  Awarded 
contract 

US  Army 

August  1992 

Capability  Evaluation  (SCE) 

Level  3;  Awarded 
contract 

us  Navy 

October  1992 

Process  Assessment  (SPA) 

Level  3 

Internal,  multiple  divisions 

November  1992 

Process  Assessment  (SPA) 

Level  1-2 

Internal  pilot  using  Software 
CMMvl.O 

October  1994 

Capability  Evaluation  (SCE) 

Unknown;  Awarded 
contract 

US  Army 

February  1995 

Capability  Evaluation  (SCE) 

Unknown;  No  con¬ 
tract 

US  Air  Force 

June  1995 

Process  Assessment  (CB  A IPI) 

Level  4 

External  with  three  author¬ 
ized  Lead  Assessors 

February  1999 

Process  Assessment  (CB  A  IPI) 

Level  5 

External  with  three  author¬ 
ized  Lead  Assessors 

4.12.1  ROI  and  Improvement  Trend  Data 

The  3.5  person  group  responsible  for  process  improvement  at  LM  NE&SS-USS  Manassas  is 
concerned  with  all  processes.  Therefore,  it  is  difficult  to  cleanly  define  the  amount  invested  in 
software  process  improvement.  However,  the  results  of  software  process  improvement  are 
easier  to  measure  and  determine.  Some  of  the  results  are  delineated  below: 

1.  350%  increase  in  software  productivity 

2.  variance  of  both  actual  cost  and  schedule  about  target  values  decreased 
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3.  87%  decrease  in  the  number  of  delivered  defects  (graphic  shown  at  1999  High  Maturity 

Workshop) 


4.1 2.2  Barriers  to  Achieving  High  Maturity 

The  biggest  barrier  was  gaining  an  understanding  of  what  is  truly  meant  by  the  two  Level  4 
Key  Process  Areas  (KPAs).  The  first  impression  one  comes  away  with  after  reading  Quantita¬ 
tive  Process  Management  is  a  description  of  mature  management  by  metrics.  The  intended 
rigor  of  statistical  tools  and  particularly  statistical  process  control  (control  charts)  does  not 
come  through. 

Once  this  understanding  has  been  achieved,  the  next  barrier  becomes  one  of  determining 
which  (sub)processes  should  be  measured  and  controlled.  All  too  often  companies  measure 
what  is  easiest  or  convenient  and  not  what  is  important.  As  with  most  changes,  you  have  to 
have  both  the  will  to  change  and  a  person  or  group  willing  to  work  to  get  the  change  made 
part  of  the  organization. 

4.12.3  Unique  or  Distinguishing  Practices 

For  many  years,  LM  NE&SS-USS  Manassas  has  used  a  software  reliability  growth  model 
(SRGM)  to  produce  a  latent  defect  projection  (LDP).  The  SRGM  is  a  home  grown  tool  based 
on  how  software  engineering  is  performed  at  LM  NE&SS-USS  Manassas.  The  LDP  is  one 
tool  to  help  determine  the  effectiveness  of  our  software  processes. 

Data  from  all  inspections  are  rolled  up  to  provide  a  composite  profile  for  a  program.  From 
this  data,  an  SRGM  is  run  to  provide  a  LDP.  The  LDP  is  an  estimate  of  the  software  quality 
(number  of  defects)  of  the  product  after  delivery.  The  LDP  is  an  early  indicator  of  possible 
problems  with  either  a  product  or  process(es). 

4.12.4  People  and  Cultural  Issues 

All  new  hires,  college  or  professional,  receive  a  core  set  of  required  training.  Even  with  this 
training,  we  have  found  that  some  new  hires  coming  from  a  less  structured  environment  may 
resist  the  imposition  of  processes.  Usually,  once  they  have  observed  the  benefit  gained  from 
process  adherence,  these  same  people  become  process  advocates. 

4.12.5  Continuing  Improvement 

Having  achieved  Level  5,  we  see  our  next  jobs  as  two  parts:  maintain  and  improve  our  soft¬ 
ware  processes  and  extend  process  maturity  to  the  related  engineering  disciplines,  especially 
systems  engineering.  Toward  that  end,  LM  NE&SS-USS  Manassas  is  moving  toward  the 
CMMI  model.  The  biggest  barrier  currently  seen  is  the  existence  of  the  two  representations  of 
the  model. 
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4.12.6  Summary 

The  effects  on  the  organization  vary  depending  on  the  view  one  takes.  If  you  look  at  the  or¬ 
ganization  as  being  made  up  of  groups  of  engineers,  the  effect  is  almost  nonexistent.  Most 
engineers  are  simply  doing  their  jobs  and  not  worrying  about  the  maturity  of  the  processes 
they  are  following. 

To  the  organization  as  a  whole,  being  a  high  maturity  organization  is  a  very  nice  recognition 
of  the  many  years  of  continual  process  improvement. 

4.13  Mastek  Limited,  Mumbai,  India 

5 

September  2000 
Ronald  Radice 
Rajshekharan  P 
www.mastek.com 

Total  1100  professionals,  of  which  offshore  professionals  are  ap¬ 
proximately  600 

20-30  people  for  a  typical  project 
Product:  Supply-chain  management  (GOLDMINE) 

Domain-specific:  Insurance,  Securities,  Education,  Finance,  Tele¬ 
com 

Platform: 

Backend:  Oracle  SQL  server,  Informix,  Ingres,  Sybase 
Front  end:  VB,  ASP,  Java,  C,  VC++,  C++,  Open  Road 
Tools:  Oracle  Apps,  CRM,  Site-server,  Work  flow 
Mobile  application.  Wireless  application 

ISO  certification  since  1994.  Assessed  at  Software  CMM  Level  4  in  1999.  Assessed  at  People 
CMM  Level  3  in  2000. 

4.13.1  ROi  and  Improvement  Trend  Data 

The  performance  of  our  improvement  program  is  measured  through  the  following  parame¬ 
ters: 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor(s) 

Point  of  Contact 
Web  Page 

Size  of  the  Organization 

Typical  Program  Size 

Primary  Application  Do- 
main(s) 


•  overall  customer  satisfaction 

•  quality  (%  rework  and  %  defect  leakage  to  customer) 

•  efficiency  (estimation  efficiency  and  productivity) 

•  timeliness  (delivery  slippage) 
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Some  graphical  representation  of  the  trends  on  these  above  parameters  is  given  below  for  the 
time  period  of  one  year. 


Figure  18:  Mastek  Customer  Satisfaction 


Figure  19:  Mastek  Rework  Trend 
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Defect  Leakage  Trend  -  Jan  2000  -  Dec  2000 


Figure  20:  Mastek  Defect  Leakage  Trend 


Estimation  Efficiency  Trend  Jan  2000  •  Dec  2000 


'%Estimation  Efficiency 


Figure  21:  Mastek  Estimation  Efficiency  Trend 
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Figure  22:  Mastek  Productivity  Trend 


Figure  23:  Mastek  Delivery  Trend 

4.1 3.2  Barriers  to  Achieving  High  Maturity 

Main  difficulties  faced  while  practicing  Levels  4  and  5  were 
1.  Implementing  Metrics  Program,  in  terms  of 

•  finalizing  required  measurement  parameters  at  different  levels  of  review 

•  data  collection  methodology 

•  appreciating  the  benefit  of  data  analysis  and  applying  SPC 

Action  taken  for  smooth  transition 

•  finalization  of  metrics  program  through  Quality  Improvement  Team  (QIT)  consist¬ 
ing  of  experts  from  process  group,  corporate  functions  and  project  managers  and 
heads  from  line  functions 
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•  automation  of  data  collection  and  reporting  through  project  tracking  tools  (ARTS) 
developed  in-house 

•  training  of  SPC  at  different  levels,  such  as  process  group,  senior  management,  pro¬ 
ject  managers,  and  team  members 

•  proper  stratification  of  data  for  analysis  and  defining  measurement  parameters  dis¬ 
tinctly  as  review  parameters  and  controlling  parameters  through  SPC 

2.  Implementing  Process  Change  Management  (PCM)  in  the  organization,  in  terms  of 

•  creating  the  awareness  and  involving  all  practitioners  in  PCM  drive 

•  handling  suggestions  from  different  directions  and  implementing  in  the  organiza¬ 
tional  procedure  and  guidelines 

Action  taken  for  smooth  transition 

•  defining  proper  work  flows  for  process  improvement  suggestions  coming  from  dif¬ 
ferent  directions 

•  campaigning  on  PCM,  regular  updates  with  implementation  status,  and  rewarding 
practitioners  for  good  suggestions 


Figure  24:  Mastek  and  Continuous  Improvement 

4.13.3  Unique  or  Distinguishing  Practices 

Three  important  contributors  and  characteristics  of  the  company  in  the  current  situation  are: 
1.  Evolution  of  SQA  role  with  higher  maturity  level 

•  SQA’s  activity  and  audits  mainly  focus  on  the  data  validation,  project  review  with 
quantitative  data  analysis 

•  SQAs  play  a  major  role  in  defect  prevention,  early  detection  through  peer  reviews, 
and  helping  practitioners  for  contributing  in  process  improvement 
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•  automation  of  routine  SQA  activities,  so  that  SQA  can  devote  more  time  in  effective 
process  improvement 

2.  Process  change  suggestion  at  almost  all  levels  are  data  driven 

•  at  project  level,  SQAs  and  project  members  analyze  project  performance  data  and 
come  out  with  project  action  items  and  organization-wide  process  changes 

•  at  organization  level,  corporate  process  groups  analyze  trends  across  projects,  calcu¬ 
late  capability  bands,  and  come  out  with  process  change  suggestions 

3.  Project  management  and  review  is  fully  data  driven  with  the  help  of  on-line  Dash  Board 
giving  charts  for  all  measurement  parameters. 

•  Mastek  uses  e-PMO  as  a  project  management  tool,  which  is  an  in-house  customized 
solution.  This  solution  is  Web-enabled  and  provides  support  for  various  activities  in 
project  management  like  request  or  service  call  registration,  work  planning,  time 
and  defect  tracking,  and  various  MIS  reports  for  project’s  progress  review.  These  re¬ 
ports  are  available  for  internal  project  mangers  as  well  as  the  customer’s  review. 


4.13.4  People  and  Cultural  Issues 

After  attaining  high  maturity  in  the  existing  processes,  the  main  challenges  are  to 

•  maintain  consistent  ROI 

•  align  and  educate  customers,  who  are  at  lower  maturity  levels 

4.13.5  Continuing  Improvement 

Current  improvement  objectives  are 

•  productivity  improvement  at  individual  level 

•  fine-tuning  processes  and  guidelines  for  very  small  deliveries  /  projects 

•  carrying  on  process  improvement  programs  through  self-managed  teams 


4.13.6  Summary 

The  benefits  seen  in  the  organization  can  be  categorized  as 
Internal  benefits: 

1 .  a  disciplined  and  consistent  way  of  managing  projects 

2.  smooth  transitioning  of  newcomers  into  organization  and  projects 

3.  predictability  of  deliverables  improved  substantially 

4.  continuously  evolving  processes  and  ability  to  adapt  them  quickly  across  similar  pro¬ 
jects 

5.  customer  confidence  and  increase  in  businesses  share  from  existing  customers.  Repeat 
business  grew  to  83%  from  76%  last  year. 

6.  process  improvement  ownership  moved  from  SEPG  to  practitioners 
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Business  and  Customer  benefits: 

1 .  ability  to  meet  demands  of  a  dynamic  industry 

2.  transparency  for  customer  through  integrated  project  management  tool  —  e-PMO 

3.  higher  return  to  customer  due  to  improved  productivity,  reduction  in  waste  and  meeting 
the  timelines 

4.14  Applied  R&D  Center,  Asia  Pacific  Telecom  Carrier 
Solutions  Group,  Motorola,  Beijing,  China 

Maturity  Level  5 

Date  of  Assessment  December  2000 

Lead  Assessor(s)  Motorola  QSR  SS 10  (Quality  System  Review  Subsystem  10) 

Point  of  Contact  Graham  Hu,  qchl422@email.mot.com 

Size  of  the  Organization  250  in  2001 

Typical  Program  Size  20  engineers  per  typical  project 

40,000  lines  of  code  per  typical  program 

Primary  Application  Domain(s)  Systems  software,  used  to  control  physical  devices 

•  Assessed  at  maturity  level  4  in  1999,  Motorola  QSR  Subsystem  10 

•  Assessed  at  maturity  level  5  in  2000,  Motorola  QSR  Subsystem  10 

•  Strong  SPI  sponsorship  from  senior  management 

•  SPI  participation  is  organization-wide 

•  Conduct  the  SPI  effort  based  on  the  data  analysis 

•  Use  of  automated  software  process  measurement  system 

•  Use  of  statistical  process  control 

•  Quality  prediction  and  control 

•  Use  of  ODC 

•  Extensive  process  training  program 
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4.15  Motorola  China  Software  Center,  Beijing/Nanjing/ 
Chengdu,  China 

5 

September  2000 
Dan  Weinberger 
Patricia  McNair 

John  Junan  Yu,  A14810@email.mot.com 

350 

Number  of  people  per  typical  project:  15-30  persons 

Number  of  lines  of  code  or  function  points  per  typical  program:  20.5 
Function  Points  per  staff  month  or  6.56  KAELOC  /  staff  month^ 

Wireless  communication 

-  GSM  /  CDMA  /  3G  infrastructure  and  subscriber  software 
developmentCommon  test  platform  (SDL,  MSC,  TTCN, 
SW  simulation)  and  test  automation 

Embedded  system  and  software  tools 

-  Real  time  OS  and  embedded  system 

-  Device  drivers 

-  Voice  recognition  and  synthesis,  digital  audio  /  video 
~  DSP  applications,  simulation  and  algorithm 

-  Compiler,  IDE  tools,  modeling  and  simulation  tools 

-  Audio  /  video  processing 

Network  management 

-  CDMA  /  GSM  network  management  system 
Internet  and  eCommerce 

Established  in  1993,  Software  Center,  Motorola  China  (SCMC)  is  an  integral  part  of  Mo¬ 
torola  Global  Software  Group.  SCMC  is  the  first  organization  in  mainland  of  China  with 
business  development  based  on  the  Capability  Maturity  Model  for  Software  (Software 
CMM).  SCMC  is  the  first  SEI  Software  CMM  Level  5  organization  in  Great  China.  There  are 
three  locations  in  Great  China;  they  are  Beijing,  Nanjing,  and  Chengdu. 

Motorola  China  Software  Center  was  assessed  as 

•  Software  CMM  Level  5  in  September  2000  by  SEI-authorized  Lead  Assessors  in  CB  A 
IPI 

•  Software  CMM  Level  4  in  January  1999  by  Motorola  assessors  in  CBA  IPI  /  QSR  SSIO 

4.15.1  ROI  and  improvement  Trend  Data 

In  SCMC,  there  are  more  than  5%  of  total  engineering  staff  working  in  the  area  of  software 
process  improvement,  quality  assurance,  new  technology  induction  and  software  engineering 

^  We  use  1  function  points  =  320  lines  of  assembly  code. 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor(s) 

Point  of  Contact 
Web  Page 

Size  of  the  Organization 
Typical  Program  Size 

Primary  Application  Domain(s) 
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training.  There  is  a  dedicated  software  quality  engineer  in  each  project  (product)  develop¬ 
ment. 

Total  customer  satisfaction  in  past  two  years:  9  points  (range  from  0-10,  6=good,  8=very 
good,  9=excellent,  10=world  class) 

Productivity  in  typical  domain:  6.56  KAELOC  /  staff  month,  which  is  much  higher  than  the 
industry  average 

Post-release  defects:  0.0032  defect  /  KAELOC  (6  sigma) 

In-process  fault  is  under  control  limit 

4.15.2  Barriers  to  Achieving  High  Maturity 

The  following  barriers  were  overcome  during  our  Software  CMM  Level  5  implementation 

•  Process  culture  building,  how  to  balance  the  long-term  investment  and  short-term  benefit. 
The  local  (Chinese)  culture  sometimes  is  the  barrier.  We  set  up  the  international  culture 
step  by  step. 

•  We  introduced  new  measurement  technology,  new  tools  and  environment.  We  set  up  a 
Software  CMM  expert  team  to  lead  the  development  team  from  Level  3  to  Level  4  and 
Level  5. 

•  We  got  strong  senior  management  commitment 

4.15.3  Unique  or  Distinguishing  Practices 

Two  of  our  distinguishing  practices  are  culture  building  and  strong  commitment  from  senior 
management. 

4.15.4  People  and  Cultural  Issues 

There  are  a  lot  opportunities  from  outside  to  experienced  Software  CMM  practitioners  in 
China;  how  to  keep  them  is  a  big  issue. 

Chinese  culture  and  English  language  are  two  big  barriers  in  China. 

Any  changes  in  process  and  technology  should  be  business  oriented. 

4.15.5  Continuing  Improvement 

Continue  to  improve  our  process  and  introduce  new  technology  to  meet  our  business  needs. 
Fine  tune  our  current  process  and  combine  CMMI  into  our  current  practice. 
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Because  many  engineers  in  China  are  emigrating  to  U.S.  and  Canada,  how  to  train  their  re¬ 
placements  is  a  key  issue. 

4.15.6  Summary 

Levels  4  and  5  make  high  expectation  from  customer. 

Levels  4  and  5  mean  high  quality,  on-time  delivery,  high  cost  efficiency. 

Level  4  and  5  organizations  can  fine-tune  their  organization  infrastructure,  management  style. 
Most  of  their  activities  can  be  presented  quantitatively. 


4.16  United  Space  Alliance,  Houston,  TX 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor(s) 

Point  of  Contact 
Web  Page 

Size  of  the  Organization 
Typical  Program  Size 

Primary  Application  Domain(s) 


5 

November  1989 
Donald  Sova 

Julie  Barnard,  Julie.r.barnard@usahq.unitedspacealliance.com 
www.unitedspacealliance.com 

300  software  professionals  (full-time  employees,  not  including  tem¬ 
porary  staff) 

100  people  per  typical  project 

500,000  lines  of  code  or  function  points  per  typical  program 
Real  Time  Applications 


The  Space  Shuttle  Onboard  Software  project  is  a  part  of  the  United  Space  Alliance  Limited 
Liability  Company  based  in  Houston,  TX.  The  project  was  previously  a  part  of  the  IBM,  Lo¬ 
ral,  and  Lockheed  Martin  corporations  before  joining  United  Space  Alliance  on  July  4,  1998. 
The  project  (under  IBM  at  the  time)  was  assessed  at  Level  5  in  November  of  1989  by  an  SEI- 
trained  NASA  team,  led  by  Donald  Sova  of  NASA  Headquarters  Code  Q. 

The  project  was  initially  registered  to  ISO  9001  in  1994  and  has  maintained  its  ISO  9001 
compliance  for  the  last  six  years.  Additionally  the  project  was  awarded  the  IBM  Best  Soft¬ 
ware  Laboratory  at  the  time  IBM  was  using  the  Malcolm  Baldrige  National  Quality  Award 
criteria  as  an  internal  assessment  method. 


The  Space  Shuttle  Software  project  is  responsible  for  the  software  executed  onboard  during 
all  phases  of  the  Shuttle  flight.  In  addition,  the  project  also  maintains  the  application  tools 
that  support  configuration  management,  test  and  simulation,  verification,  and  flight  recon¬ 
figuration. 

To  satisfy  NASA’s  requirement  for  software  that  meets  the  highest  safety  and  reliability  stan¬ 
dards,  the  project  evolved  a  software  process  that  yields  a  highly  predictable  quality  result. 
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By  executing  the  process  faithfully  to  specified  process  standards,  the  software  produced  by 
the  process  is  predictably  near  zero  defects.  The  processes  have  been  modified  and  refined 
over  two  decades  to  create  an  optimized  process  suitable  for  producing  software  meeting  the 
desired  expectations. 

The  project  size  fluctuates  around  250-300  people.  The  project  resides  within  a  larger  soft¬ 
ware  organization  of  approximately  600  people  that  includes  a  major  subcontractor  assessed 
at  Software  CMM  Level  5  in  October  1999. 

4.16.1  ROI  and  Improvement  Trend  Data 

NASA’s  overriding  concern  is  for  the  quality  and  safety  of  the  Shuttle  software  at  the  time  it 
is  certified  ready  to  fly  a  specific  mission.  Accordingly,  the  customer  specifies  defect  density 
thresholds  applicable  to  the  software  at  this  delivery  milestone.  One  of  the  objectives  of  proc¬ 
ess  improvement  is  to  minimize  the  remaining  errors  in  the  delivered  software. 

During  the  period  of  process  improvement  leading  up  to  the  Software  CMM  Level  5  rating, 
the  defect  density  improved  significantly.  This  demonstrated  a  correlation  between  the  matu¬ 
ration  of  the  processes  and  the  improvement  in  delivered  product  quality.  As  the  process  ac¬ 
tivity  turned  more  to  the  nature  of  process  refinements,  rather  than  massive  process  over¬ 
hauls,  the  defect  density  continued  to  improve  but  at  a  slower  pace.  Excellent  product  quality 
has  been  sustained  in  a  predictable  zone  for  the  number  of  years  since. 

4.16.2  Barriers  to  Achieving  High  Maturity 

The  project  began  early  with  high  maturity  practices  due  to  the  life  criticality  of  the  software 
and  the  quality  expectations  levied  by  the  customer.  A  driving  expectation  of  zero  defect 
software  made  the  move  to  Level  5  a  natural  path. 

Because  the  project  was  assessed  at  a  high  maturity  level  some  time  ago,  the  current  barriers 
facing  the  project  relate  more  to  the  sustainment  of  high  maturity  at  an  organizational  level, 
rather  than  to  achieving  high  maturity. 

One  of  the  challenges  facing  the  project  over  the  more  than  ten-year  period  since  its  Software 
CMM  Level  5  assessment  has  been  to  sustain  its  practices  across  the  company  changes  and 
reorganizations.  Through  corporate  transitions,  reorganizational  growing  pains,  expansion, 
etc.,  the  project  has  encountered  a  need  to  periodically  assess  itself  against  the  Software 
CMM.  These  internal  CMM-based  assessments  are  done  to  ensure  there  is  no  backsliding  of 
existing  practices,  to  fold  newer  parts  of  the  organization  into  high  maturity  practices,  and  to 
ensure  continuity  in  infrastructure  across  organizational  transitions.  By  leaving  intact  the 
basic  organization  structure,  core  management,  and  process  infrastructure,  the  best  practices 
have  continued  successfully  across  the  corporate  boundaries. 
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The  organization  is  expanding  to  include  groups  who  were  not  a  part  of  the  original  project 
and  who  are  striving  for  improving  their  process  maturity.  Integrating  lower  maturity  groups 
into  an  existing  project  steeped  in  tradition,  a  high  maturity  mentality,  and  a  culture  focused 
on  zero  defects  has  been  the  latest  challenge.  Groups  from  a  variety  of  corporate  genealogies 
that  are  accustomed  to  diverse  cultures,  methods,  toolsets,  and  terminology  are  being  joined 
together  to  blend  the  best  of  each  area. 

4.16.3  Unique  or  Distinguishing  Practices 

Some  of  the  distinguishing  practices  the  project  is  recognized  for  include: 

•  rigorous  application  of  forma!  inspections,  including  software  requirements  inspections 

•  in-depth  defect  prevention  process,  including  robust  root  cause  analysis  performed  on 
each  software  defect  that  is  a  detailed  examination  to  determine  causes  for  process  es¬ 
capes 

•  matrixed  program  management  and  control  board  structure 

•  long-term  continuous  process  improvement 

•  reliability  analysis  and  prediction  modeling  based  upon  historical  data 

•  senior  manager  reviews  of  project  key  processes  (e.g.,  inspection,  requirements  evalua¬ 
tion,  development  test,  etc.)  to  examine  process  escapes,  process  changes,  and  process 
metrics 

•  use  of  process  enactment  type  tools  to  facilitate  process  handoffs 

4.16.4  People  and  Cultural  Issues 

Aside  from  some  expected  fallout  of  personnel  across  three  corporate  transitions,  the  project 
has  benefited  from  the  retention  of  core  skills  in  key  functional  areas.  There  has  been  a  fairly 
large  amount  of  hiring  in  the  past  couple  of  years,  which  is  partially  due  to  expanding  work 
opportunities.  The  attrition  rate  is  typically  quite  low,  as  many  employees  remain  long  term 
on  the  project. 

The  mentoring  program  and  new  hire  orientation  have  been  used  for  many  years  to  indoctri¬ 
nate  new  employees  into  the  complex  software  domain.  Due  to  the  amount  of  time  it  takes  to 
prepare  an  employee  before  taking  on  active  assignments,  the  project  relies  on  these  pro¬ 
grams  heavily.  The  mentoring  program  includes  a  documented  formal  process  that  establishes 
assigned  mentor  /  mentee  relationships  and  conducts  surveys  periodically  to  monitor  pro¬ 
gress.  It  allows  for  a  no-fault  termination  of  mismatched  mentor  /  mentee  pairs  in  situations 
where  the  assignments  just  did  not  work  out.  The  new  hire  orientation  includes  general  pro¬ 
ject  overviews,  detailed  process  presentations  by  owners  of  each  process,  domain  specific 
training,  and  other  required  training  specific  to  the  organization.  This  supplements  the  com¬ 
pany  provided  new  hire  topics. 
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The  high  maturity  culture  is  fostered  by  ongoing  and  visible  senior  management  commit¬ 
ment,  involvement,  and  sponsorship.  The  senior  manager  conducts  periodic  project  all  hands 
meetings  including  VIPs,  reinforces  accomplishments  and  achievements  with  ample  recogni¬ 
tion,  and  communicates  strategic  direction  for  the  organization — outlining  future  opportuni¬ 
ties  for  project  growth.  In  addition,  process  team  empowerment  and  employee  ownership  of 
processes  is  the  norm. 

Recently,  a  prospective  employee  approached  one  of  the  project  managers  and  inquired  about 
the  background  of  our  high  maturity  organization.  He  indicated  that  he  was  tired  of  dealing 
with  a  Level  1  environment  (e.g.,  working  extensive  overtime  and  always  being  schedule 
driven).  He  had  come  to  the  project  looking  for  an  oasis  of  work  refreshment. 

4.16.5  Continuing  Improvement 

The  project  has  been  working  on  process  improvement  for  over  two  decades,  beginning  with 
major  changes  in  fundamental  process  approaches  and  then  evolving  to  refinements  of  proc¬ 
esses  with  less  extensive  changes.  A  current  focus  is  on  research-oriented  prototyping  of 
techniques  for  potential  process  improvement  identification.  Within  the  project  there  are 
planned  research  activities  to  investigate  the  applicability  of  certain  techniques  /  models  to 
improve  prediction  of  remaining  defects  at  delivery.  Because  this  ties  closely  to  the  objec¬ 
tives  of  end  product  delivered  quality,  the  project  continues  to  pursue  improved  methods  for 
examining  data,  early  indicators,  and  analysis  techniques  that  will  provide  varied  insight  into 
its  existing  processes. 

Future  improvement  activities  also  include  transferring  mature  practices  from  within  the  pro¬ 
ject  to  other  parts  of  the  company.  The  corporate  software  process  owner  and  company  SEPG 
serve  as  the  primary  mechanism  for  advancing  the  software  process  maturity  of  the  overall 
company  software  process.  The  project  is  contributing  its  process  expertise  to  review  soft¬ 
ware  processes  in  other  parts  of  company  and  to  aid  in  the  development  of  mature  processes. 

The  corporate  software  process  goals  include  plans  for  formal  company-wide  assessment 
against  Software  CMM  vl.l.  This  assessment  will  include  at  least  some  part  of  the  current 
flight  software  organization.  There  is  a  long-term  expectation  of  evolving  the  corporate  soft¬ 
ware  and  systems  engineering  processes  towards  CMMI.  An  assessment  against  CMMI  of 
the  software  process  would  be  a  longer-term  objective. 

Similarly,  the  use  of  ISO  9001  will  evolve  from  the  1994  version  of  the  standard  to  the  more 
current  version.  The  project  will  be  assessed  against  the  revised  9001  at  the  time  the  company 
registration  scope  is  modified  to  step  up  to  the  newer  version  of  the  approved  standard. 

The  NASA  Software  Working  Group  is  developing  an  agency-wide  software  standard  that 
incorporates  the  requirements  of  lEEE/EIA  12207  for  development  of  mission  critical  soft¬ 
ware.  As  this  standard  is  deployed  to  NASA  projects,  it  is  likely  that  the  applicable  software 
projects  will  also  assess  compliance  to  12207. 
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4.16.6  Summary 

A  built-in  part  of  a  mature  process  is  to  change  the  process  and  assure  continuous  improve¬ 
ment.  Even  a  project  that  has  operated  over  two  decades  with  success  has  continued  to  im¬ 
prove  and  change.  Skilled  people  and  a  continuously  improving  process  produce  quality 
products.  Process  changes  are  identified  from  both  detailed  introspective  analysis  and  metric 
analysis  with  a  goal  of  process  optimization.  Process  changes  are  prioritized  then  tracked  by 
program  management  to  ensure  attention  is  paid  to  critical  changes.  Appropriate  process 
changes  are  tied  to  customer  commitments.  Modeled  and  predicted  product  quality  and  reli¬ 
ability  are  compared  to  actual  quality  and  reliability  for  tracking  overall  product  objectives. 

There  are  some  mandatory  organizational  attitudes  that  prevail  in  the  high  maturity  organiza¬ 
tion: 

•  management  and  employee  obsession 

•  discipline 

•  perfection  expectation 

•  long-term  commitment 

•  employee  empowerment  and  ownership 


4.17  Philips  Software  Centre,  Bangalore,  India 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor(s) 

Point  of  Contact 
Web  Page 

Size  of  the  Organization 
Typical  Program  Size 


Primary  Application  Domain(s) 


5 

July  2000 
Richard  Knudson 

Bob  Hoekstra,  bob.hoekstra@philips.com 
http://www.philipssoftwarecentre.com 
About  500+  during  assessment 

About  8-10  persons  /  project  (there  are  bigger  as  well  as  smaller  projects 
than  this).  In  PSC,  “Program”  is  used  to  mean  LOB,  and  these  are  50-100 
persons  in  size. 

Embedded  software  used  in  TVs,  DVDs,  adhering  to  internal  Philips  stan¬ 
dards. 


Philips  Software  Centre  is  a  fully  owned  subsidiary  of  Royal  Philips  Electronics  NV,  Nether¬ 
lands.  This  company  was  inaugurated  in  September  1996.  PSC,  Bangalore  was  assessed  at 
Software  CMM  Level  3  in  September  1997  (as  per  Philips  assessment  method)  and  ISO 
9001/TickIT  Certified  in  December  1997. 

The  journey  towards  Software  CMM  Level  5  began  with  a  top  management  workshop.  That 
was  followed  by  a  documentation  gap  analysis  to  enhance  process  definition,  a  Web-based 
abridged  assessment,  and  two  five-day  mini  assessments  prior  to  the  CBA IPI  in  June  2000. 
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4.17.1  ROI  and  Improvement  Trend  Data 

The  Business  Plan  for  PSC  is  translated  into  long  term  Quality  Improvement  Plan  (Macro 
plan  for  two  years,  updated  every  months)  and  SPI  Plan  derived  from  the  QIP  (Micro  plan  for 
six  months,  updated  bi-monthly).  PSC  has  a  dedicated  Corporate  Quality  Department  that 
coordinates  all  organization-level  improvement  activities.  The  company’s  software  process 
improvement  journey  was  tracked  for  all  expenses  incurred  including  opportunity  cost  of 
time  spent. 

The  target  setting  process  in  PSC’s  quality  journey  was  evolutionary  in  nature.  The  earlier 
targets  were  process  compliance  targets  and  subsequently  we  have  set  three  product-related 
targets.  These  targets  are  listed  below. 

1.  Ensuring  process  maturity  at  Level  5.  This  included  enhancing  documented  processes  to 
meet  Level  5  requirements  (process  definition  target)  and  deploying  documented  proc¬ 
esses  to  meet  Software  CMM  Level  5  requirements  (deployment  target). 

2.  Putting  in  place  a  measurement  program  that  gave  us  statistically  derived  limits  for  all 
key  performance  indicators  (Pis)  and  tracking  performance  against  these  Pis.  The  per¬ 
formance  indicators  we  used  were  -  Time  Slip,  Effort  Slip,  Cost  of  Non  Quality,  Review 
Effectiveness  and  Pre-release  and  Post-release  defect  levels.  Despite  a  huge  influx  of 
new  staff  and  creation  of  several  new  Product  Divisions,  our  performance  during  the 
year  2000  has  improved  for  review  effectiveness  and  CONQ. 

3.  Achieving  a  10-fold  reduction  in  post  release  defect  levels  at  PSC,  starting  May-June 

2000. 


Figure  25:  Philips  Time  Slip  Trend 
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Figure  26:  Philips  Effort  Slip  Trend 
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Figure  28:  Philips  Review  Effectiveness  Trend 


Figure  29:  Philips  Post-Release  Defect  Density  Trend 
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Customer  Satisfaction  Trends  -  PMS,  MCE  &  PS 


1998  1999  2000 

Year 


Figure  30:  Philips  Customer  Satisfaction  Trends 

4.17.2  Barriers  to  Achieving  High  Maturity 

The  five  barriers  we  encountered  in  moving  up  on  the  maturity  rating  were  as  follows: 

1.  We  had  a  large  percentage  of  new  staff  that  were  not  trained  to  follow  our  processes.  So 
we  had  issues  at  Levels  2  and  3  even  while  we  were  working  on  demonstrating  capabil¬ 
ity  at  Levels  4  and  5.  Inducting  new  staff  takes  time.  To  reduce  the  time  required  for  new 
staff  to  follow  our  processes  we  organized: 

•  process  training  for  new  entrants 

•  refresher  training  for  old  staff  on  processes  that  underwent  significant  changes 

•  more  forums  for  sharing  of  best  practices  within  the  company 

•  the  annual  calendar  for  core,  process  and  technical  training  is  published  for  staff  to 
plan  their  training 

2.  A  concern  among  senior  project  managers  that  a  Software  CMM  Level  4  or  5  rating 
would  raise  customer  expectations,  and  we  may  not  be  able  to  fulfill  these  expectations 
since  performance  may  take  some  time  to  catch  up.  This  was  addressed  by  reviewing 
available  data  to  see  what  the  trends  indicated  in  terms  of  performance.  We  also  re¬ 
viewed  customer  feedback  scores  and  trends  over  the  last  few  years.  The  reviews  indi¬ 
cated  that  in  most  cases  we  had  improved  our  performance  and  were  being  rated  better 
by  our  customers  too. 

3.  PSC,  Bangalore  is  the  first  company  in  all  of  Philips  to  be  assessed  at  Level  5.  As  of 
March  2001,  out  of  92  software  groups  in  Philips,  3  are  at  Level  3,  24  at  Level  2,  and  the 
rest  at  Level  1  (data  taken  from  SPI  Steering  committee  page).  Since  PSC,  Bangalore 
services  only  Philips,  this  difference  in  process  maturity  led  “customer”  software  or¬ 
ganizations  questioning  many  of  our  processes  and  process  initiatives.  This  barrier  was 
addressed  by  giving  more  flexibility  to  our  project  managers  to  define  the  PDSP  through 
a  software  process  handbook.  To  ensure  that  all  KPAs  were  complied  with,  this  process 
handbook  needs  to  be  signed  off  by  the  Corporate  Quality  department. 

4.  We  also  faced  an  issue  of  buy-in  while  treating  all  PSC  as  one  entity  while  trying  to  es¬ 
tablish  statistical  norms.  As  each  Line-of-Business  has  separate  customers  who  work 
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under  different  business  circumstances,  the  time  slip  and  effort  slip  norms  vary  across 
LOBs.  Our  initial  approach  of  bringing  all  these  together  as  the  same  process  was  fol¬ 
lowed  did  not  bear  fruit,  and  we  could  not  convince  project  teams  to  follow  these  norms. 
We  discontinued  this  approach  and  treated  each  LOB  as  a  separate  entity  and  the  buy-in 
increased  significantly. 

5.  Tight  coupling  of  our  LOBs  /  PDs  with  customer  organization  meant  that  the  scope  of  a 
KPA  like  TCM  was  difficult  to  define  since  technology  /  tools  are  sometimes  enforced 
by  customer  organization.  We  worked  around  this  by  defining  our  TCM  process  to  ad¬ 
dress  evaluation  and  deployment  in  cases  where  the  customer  did  not  mandate  the  tool. 


4.17.3  Unique  or  Distinguishing  Practices 

The  unique  things  we  are  doing  today  include  the  following: 

•  Collecting  a  lot  more  data  than  we  did  at  Level  3.  We’re  collecting  data  on  “hard”  meas¬ 
ures  like  time  and  effort  slip,  cost  of  non-quality,  etc.,  and  “soft”  issues  like  “competency 
ratio.” 

•  Our  norms  today  are  based  on  statistically  derived  limits  of  XbarR  Charts.  Earlier  the 
norms  were  set  by  top  management  and  these  norms  were  not  statistically  derived. 

•  We  understand  the  key  (root)  causes  of  our  defects  and  are  taking  systemic  actions  to  re¬ 
duce  defect  levels. 

•  We  maintain  a  Tools  Repository  on  available  software  tools  and  their  benefits  so  new 
projects  can  use  this  information  for  planning.  The  software  tools  in  this  repository  are 
categorized  into  the  software  engineering  processes. 

•  We  track  employee  participation  in  improvement  activities. 

4.17.4  People  and  Cultural  Issues 

Project  Managers  believing  that  being  assessed  at  Level  5  could  lead  to  greatly  enhanced  cus¬ 
tomer  expectations  that  may  not  be  met.  A  fear  that  this  could  cause  customer  dissatisfaction 
could  lead  to  some  opposition  to  a  company’s  quest  for  Level  5. 

For  institutionalization,  SPI  should  be  owned  by  all  staff.  This  is  sometimes  difficult  because 
of  project  pressures  or  because  there  may  not  be  enough  improvement  projects  for  all.  Some¬ 
times  project  teams  view  SPI  as  something  different  from  their  primary  task  of  delivering  the 
project. 

4.17.5  Continuing  improvement 

We  have  set  the  following  targets: 

•  Continue  operating  at  Level  5.  To  ensure  this  we  have  planned  quarterly  internal  assess¬ 
ments  based  on  the  Philips  Assessment  Method.  This  method,  we  believe,  gives  the  same 
results  as  CB A  EPL 
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•  Achieve  ten  times  reduction  in  post  release  defects  over  two  years.  This  transforms  itself 
into  1.8  times  reduction  every  six  months.  We  needed  to  be  at  0.8  major  defects  /  KLOC 
of  code  in  December  from  1.45  major  defects  /  KLOC  of  code  in  June.  We  reached  close 
to  1.0  major  defects  /  KLOC  of  code  so  the  improvement  was  close  to  30  %  but  not  suffi¬ 
cient.  Our  focus  in  the  months  to  come  will  be  on  defect  reduction.  We  intend  achieving 
it  through  : 

”  better  deployment  of  Defect  Prevention  process 

-  increased  automation  and  usage  of  tools 

-  increased  levels  of  technical  competency  in  PSC  staff. 

PSC,  Bangalore  is  currently  deploying  a  Business  Balanced  Score  Card  based  Software 
Measurement  Program  at  Project,  LoB,  and  PSC  Level,  with  clearly  defined  leading  indica¬ 
tors  to  give  better  visibility  and  ensure  timely  corrective  actions.  This  will  ensure  better  per¬ 
formance  on  lagging  indicators  viz.  customer  satisfaction,  post-release  defects,  etc. 

We  are  targeting  a  Philips  Business  Excellence  score  of  600  by  end  of  2001.  The  Philips 
Business  Excellence  model  is  based  on  the  EFQM  model  and  looks  at  enablers  as  well  as 
results  in  achieving  excellence.  The  enablers  include  leadership,  policy  and  strategy,  people, 
partnerships,  and  resources,  and  processes.  The  results  include  customer  results,  people  re¬ 
sults,  society  results,  and  key  performance  results. 

4.17.6  Summary 

In  PSC  Bangalore,  Levels  4  and  5  mean 

•  usage  of  statistical  norms  to  control  our  performance  on  performance  indicators  (both 
process  and  product). 

•  having  data  to  bring  about  improvements  based  on  measured  benefits.  By  this  we  mean 
that  we  have  deployed  the  concept  of  measuring  improvements  in  terms  of  monetary 
benefits. 

•  better  controls  on  defect  levels  and  a  planned  way  to  reduce  defects.  We  use  “Defect  Pre¬ 
vention”  at  the  project  level  as  well  as  LOB  Level  and  Organization  Level  and  this  gives 
indications  of  key  actions  required  to  reduce  defect  levels. 

•  bringing  technology  changes  in  a  controlled  manner  based  on  data — we  have  used  TCM 
primarily  for  increasing  tool  usage  in  our  software  creation  process. 
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4.18  Satyam  Computer  Services  Ltd.,  Hyderabad, 
India 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor(s) 

Point  of  Contact 
Web  Page 

Size  of  the  Organization 
Typical  Program  Size 


Primary  Application  Domain(s) 


5 

March  1999  for  all  Indian  development  centres 
Richard  W  Knudson 

Prabhuu  Sinha,  Prabhuu_Sinha@satyam.com 

www.satyam.com 

7000 

These  numbers  are  extremely  variable  across  Satyam’s  projects.  This 
is  because  Satyam  provides  end-to-end  software  solutions  and  ser¬ 
vices  to  many  different  companies. 

Systems  software,  used  to  control  physical  devices 
Commercial  software,  leased  or  marketed  to  external  clients 
Information  systems  software,  for  business  information 
Out-sourced  software,  developed  under  contract 
End  user  software,  private  software  for  personal  use  (shrinkwrap) 


Satyam  was  established  in  1987  and  since  then  has  been  providing  quality  software  services 
to  large  corporations  all  over  the  world.  The  company  was  one  of  the  pioneers  of  the  concept 
of  remote  software  development  using  high-speed  satellite  communication  circuits  in  India. 
Satyam  was  one  of  the  first  Indian  companies  to  acquire  a  dedicated  satellite  circuit  for  data 
communication.  Satyam  today  is  amongst  the  top  five  offshore  software  development  com¬ 
panies  in  the  country. 

Satyam  achieved  ISO  9001  certification  under  the  TickIT  scheme  in  March  1995  and  was  re¬ 
certified  to  the  same  in  September  1998.  In  January  2001,  Satyam  achieved  certification  to 
the  ISO  9001:2000  standard. 


In  March  1999,  Satyam  achieved  Software  CMM  Level  5  accreditation  for  all  its  software 
development  centers  in  India. 

At  the  national  level  in  India,  Satyam  has  won  several  quality-related  awards.  It  won  the 
Golden  Peacock  National  Training  Award  for  its  training  program  in  July  1999  and  the 
Golden  Peacock  Innovative  Service  Award  for  its  teleconference  facilities  in  January  2000. 
Satyam’s  Senior  Vice  President  for  Quality,  Prabhuu  Sinha,  won  the  QIMPRO  Silver  Stan¬ 
dard  Award  in  October  2000  for  his  leadership  of  the  quality  movement  in  Satyam. 
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4.18.1  ROI  and  Improvement  Trend  Data 

Satyam  has  observed  the  following  improvement  trends: 


Table  2:  Relative  Improvements  Observed  at  Satyam'' 


Jan  1998-Dec  1998 

Jan  1999-Dec  1999 

Jan  2000-Dec  2000 

Reduction  in  El^ort 
Variance 

40% 

No  significant  reduc- 
tion** 

41  % 

Reduction  in  Schedule 
Variance 

24% 

35  % 

30% 

Increase  in  Productiv¬ 
ity 

Baseline  not  computed 

No  significant  increase^ 

22% 

Reduction  in  Delivered 
Defect  Density 

8% 

42  % 

No  significant  reduc¬ 
tion^ 

Increase  in  Defect  Re¬ 
moval  Efficiency 

Baseline  not  computed 

12% 

6% 

4.18.2  Barriers  to  Achieving  High  Maturity 

Some  of  the  major  barriers  that  Satyam  had  to  overcome  in  achieving  Level  5  are: 

•  Satyam  has  had  a  growth  rate  of  100%  per  year  for  the  past  five  years.  This  required  a 
massive  training  and  SQA  effort  to  institutionalize  the  processes. 

•  Standard  industry  tools  for  process  automation  were  difficult  to  implement  since  they 
inherently  impose  a  software  process  that  does  not  match  with  Satyam’s  way  of  working. 
We  have  now  developed  our  own  tools  for  process  automation. 

•  Due  to  the  very  high  growth  rate,  it  took  quite  a  while  for  Satyam  to  find  suitable  and 
sufficient  staff  for  the  SQA  and  SEPG  teams.  This  was  because  the  large  numbers  re¬ 
quired  coupled  with  the  low  availability  of  such  skills  in  the  local  market. 

•  While  setting  productivity  and  quality  baselines  at  the  organization  level  and  goals  for 
each  project,  it  was  difficult  to  quantify  the  impact  of  varying  project  attributes  like  envi¬ 
ronment,  project  size  /  complexity,  team  skills  and  customer’s  process  maturity. 

•  During  short  duration  projects  (3-4  months),  it  was  difficult  to  implement  QPM  due  to 
high  schedule  pressures.  The  benefits  of  QPM  in  such  small  duration  projects  were  also 
not  tangible. 

•  It  was  difficult  to  set  goals  for  new  technology  projects.  While  Software  CMM  recom¬ 
mends  that  this  should  be  done  by  using  a  pilot  project  to  set  baselines,  this  is  difficult 
because  the  pilots  are  very  small  size  and  may  not  give  feasible  data. 


^  These  are  not  absolute  figures. 

^  Satyam  was  concentrating  on  T&M  type  of  Year  2000  work  at  that  time,  and  no  improvements 
were  observed  after  the  significant  improvement  in  Effort  Variance  in  the  previous  year. 

^  The  mean  of  the  Defect  Removal  Efficiency  has  not  changed  significantly,  but  the  3-Sigma  band 
has  narrowed  in  this  year. 
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•  Defect  prevention  learning  in  early  life-cycle  phases  often  cannot  be  used  within  the 
same  project.  It  is  used  in  other  projects  via  sharing.  Due  to  this,  the  project  does  not  see 
the  immediate  benefit  of  causal  analysis  and  DP  meetings.  This  makes  it  difficult  to  mo¬ 
tivate  project  teams  on  this. 

4.18.3  Unique  or  Distinguishing  Practices 

The  model  “Leading  the  Initiatives  for  Excellence  (LIFE)”  is  used  by  all  units  in  Satyam  for 
achieving  strategic  and  operational  excellence. 

All  business  units  and  support  units  in  Satyam  operate  as  profit  centers.  Processes  are  defined 
for  all  business  units  and  support  units,  and  periodic  customer  satisfaction  surveys  are  con¬ 
ducted  for  both  internal  and  external  customers  to  identify  areas  where  performance  can  be 
improved. 

Satyam’s  Quality  Management  System  and  Project  Knowledge  Base  are  available  on-line  to 
all  company  personnel  via  QUALIFY,  an  in-house  tool.  Another  in-house  tool,  QUANTIFY, 
automates  the  processes  for  project  planning  /  tracking  /  monitoring,  defect  tracking,  metrics 
collection  /  analysis,  defect  prevention,  and  change  management. 

Satyam  has  a  unique  quality  communications  and  rewards  program  run  by  an  Intranet-based 
tool,  the  purpose  of  which  is  to  sustain  a  vibrant  quality  culture  in  Satyam.  This  is  achieved 
by  recognizing  organizational  development  activities  in  the  area  of  quality.  The  objectives  of 
the  program  are: 

•  ensuring  organizational  wide  participation  in  Satyam’s  Quality  Journey 

•  eliciting  significant  contribution  from  practitioners,  to  Satyam’s  Quality  Journey 

•  creating  a  sense  of  ownership  in  Satyam’s  Quality  Management  System 

•  creating  a  sense  of  pride  in  using  Satyam’s  Quality  Management  System 

Satyam’s  Vision  Compass  tool  is  used  for  monitoring  key  targets  and  ongoing  activities 
across  Satyam.  It  helps  Satyam  in  virtual  operations,  senior  management  reviews,  and  proc¬ 
ess  monitoring. 

4.1 8.4  People  and  Cultural  Issues 

The  Satyam  Learning  Centre  (SLC)  facilitates  dissemination  of  knowledge  from  diverse 
fields  of  management  and  technology  for  the  benefit  of  employees  and  Satyam.  SLC  enables 
Satyam  to  learn  continuously  and  thus  be  a  learning  organization.  SLC  enables  employees  to 
understand  the  businesses  of  the  Satyam  group,  to  understand  the  core  purpose  and  core  val¬ 
ues  of  Satyam  and  to  develop  relationships  with  employees  of  various  locations. 
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The  objectives  of  SLC  are  to 


•  define  competencies  and  learning  requirements  of  various  roles  in  the  company 

•  provide  learning  inputs  to  employees  to  meet  organizational  needs  and  individual  devel¬ 
opment  needs 

•  establish  academic  linkages  with  reputed  institutions  to  leverage  on  their  competencies 
for  organization’s  development 

•  work  in  cohesion  with  Satyam  units  to  forecast  future  learning  needs  of  the  organization 

•  enable  dissemination  of  information  /  knowledge  through  library,  knowledge  repository, 
and  learning  processes 

•  induct  employee’s  family  into  Satyam’s  family 

•  provide  counseling  to  an  employee  to  overcome  workplace  /  personal  problems 


4.18.5  Continuing  Improvement 

Satyam  has  several  new  initiatives  in  the  area  of  quality.  They  are: 

•  CMMI — Satyam  is  studying  this  model.  We  plan  to  improve  our  processes  based  on  this 
model  and  go  for  an  assessment. 

•  Six  Sigma — Satyam  is  working  towards  developing  and  institutionalizing  its  own  Six 
Sigma  model. 

•  People  CMM — Satyam  is  improving  its  Human  Resources  and  Training  processes  based 
on  the  People  CMM  and  plans  to  go  for  an  assessment. 

•  Benchmarking — Satyam  is  starting  a  benchmarking  initiative  for  its  critical  processes. 

•  Satyam  Business  Excellence  award — Satyam  is  developing  a  model  for  its  own  Busi¬ 
ness  Excellence  Award.  All  units  of  Satyam  will  compete  for  this  award  and  win  recogni¬ 
tion  for  their  business  excellence. 

Satyam  is  also  improving  several  of  its  existing  processes.  They  are: 

•  Software  CMM  and  ISO — Currently,  Satyam’s  delivery  units  in  India  have  been  as¬ 
sessed/certified  on  these  models.  Satyam  is  working  towards  extending  the  scope  of  these 
models  to  its  units  all  over  the  world  and  conducting  assessments/audits  for  them. 

•  Process  performance — Satyam  is  working  towards  increasing  its  process  performance, 
particularly  in  the  areas  of  defect  prevention,  defect  detection,  and  productivity. 

•  Virtual  services — Globalization  is  one  of  Satyam’s  organizational  goals.  In  keeping  with 
this,  Satyam  is  working  towards  virtual  delivery  of  its  internal  services.  In  the  quality 
area,  this  includes  Web-enablement  of  the  quality  system,  knowledge  management 
framework,  internal  quality  audits,  and  quality-related  organizational  development  activi¬ 
ties. 
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The  barriers  currently  faced  by  Satyam  are: 


•  Growth  Areas — ^To  keep  in  pace  with  the  rapid  changes  in  the  IT  industry,  Satyam  also 
introduces  new  competencies  and  areas  of  business  at  a  rapid  pace.  This  provides  a  chal¬ 
lenge  since  new  areas  need  new  methodologies,  processes,  and  skills. 

•  Manpower — Increasing  the  staff  in  the  SEPG  and  SQA  functions  to  keep  pace  with  Sat- 
yam’s  rapid  growth  is  still  a  problem  due  to  lack  of  skilled  manpower  in  the  local  market. 


4.18.6  Summary 

The  quality  movement  in  Satyam  is  now  all-pervasive.  It  covers  not  only  software  develop¬ 
ment,  but  also  areas  like  business  and  support  processes.  Continuous  improvement  is  being 
achieved  in  process  and  product  quality  in  all  areas.  There  is  now  a  “pull”  culture  for  quality 
from  all  Satyam  units,  as  opposed  to  a  “push”  culture  that  existed  a  few  years  ago. 

Employees  of  Satyam  feel  a  sense  of  pride  in  being  part  of  a  Software  CMM  Level  5  com¬ 
pany.  They  are  highly  charged  and  committed  to  the  quality  system.  The  easy  availability  of 
data  and  other  knowledge  from  projects  has  created  a  culture  of  sharing  and  learning. 

Satyam  has  also  achieved  significant  return  on  investment  from  its  SPI  movement.  Besides 
improving  the  process  performance  and  product  quality,  achieving  Level  5  has  also  increased 
customer  confidence  in  Satyam,  resulting  in  increased  business  opportunity. 

Our  process  maturity  has  reduced  our  learning  curve,  and  thus  increased  our  capability  to 
deliver  and  continuously  improve  our  processes  even  with  a  100%  growth  rate  every  year. 

After  achieving  Software  CMM  Level  5,  we  now  have  the  confidence  to  go  for  other  world 
class  quality  related  models  like  Six  Sigma,  People  CMM,  etc. 


4.19  Siemens  Information  Systems  Limited, 
Bangalore,  India 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor(s) 

Point  of  Contact 
Web  Page 

Size  of  the  Organization 
Typical  Program  Size 

Primary  Application  Domain(s) 


4 

August  2000 
Richard  F.  Storch 

Rakesh.singh@sisl.co.in,  kathavarayan.t@sisl.co.in 

www.sislindia.com 

250 

20  staff 

40,000  to  1,40,000  LOG 

Medical  engineering,  automotive  engineering  and,  transportation 
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Siemens  Information  Systems  Limited  (SISL),  is  a  joint  venture  between  Siemens  Limited 
(India)  and  Siemens  Business  Services  of  Siemens — Information  and  Communications,  Ger¬ 
many.  It  is  involved  in  software  development,  information  technology  consulting,  and  system 
integration  services  for  the  Indian  and  global  markets  through  offices  in  Bangalore,  Calcutta, 
Chennai,  Delhi,  Mumbai,  and  Pune  in  India.  SISL  has  been  an  ISO-certified  company  since 
1996  and  again  re-certified  in  the  year  1999. 

4.19.1  ROI  and  Improvement  Trend  Data 

The  following  are  the  visible  benefits  SISL  got  as  part  of  process  improvement. 

•  estimate  efficiency  in  SISL  has  improved 

•  defect  removal  efficiency  has  improved 

•  delivered  defects  density  has  come  down 

4.19.2  Barriers  to  Achieving  High  Maturity 

Any  process  improvement  initiative  should  be  initially  tried  out  within  a  limited  organiza¬ 
tional  scope  as  any  setback  to  improvement  programs  can  cause  long-lasting  damage.  When 
the  organizational  scope  is  not  properly  defined  then  exposing  a  large  organization,  consist¬ 
ing  of  several  business  units,  to  process  changes  can  run  into  difficulties  such  as  follows: 

•  There  is  large  variation  in  process  need,  technology  deployed,  and  domain-related  priori¬ 
ties. 

•  They  are  at  different  phase  in  business  life  cycle  such  as  initiation,  stabilized,  or  phasing 
out. 

•  Managing  logistics  during  process  development  and  implementation  is  difficult. 

•  Cost  of  poor  process  or  learning  will  have  to  be  borne  by  all. 

•  Often  needs  and  priority  may  be  contradictory. 

•  Level  of  support  and  desire  to  improve  will  also  vary. 

It  makes  sense  to  consider  a  more  manageable,  at  the  same  time  significant,  organizational 
scope  for  improvement.  In  addition,  criteria  such  as  readiness  to  innovate,  resources  at  dis¬ 
posal,  and  timing  of  improvement  programs  should  also  be  considered  to  select  the  right  or¬ 
ganizational  unit.  Lastly,  the  improvement  program  should  fit  their  overall  business  schedule. 

Some  of  the  common  difficulties  encountered  to  the  process  improvement  program  are: 

•  poor  priority  for  process  development  tasks 

•  difficulty  in  attracting  and  retaining  the  right  talent 

•  top  man’s  moral  and  resource  support 
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•  best  people  not  available  or  interested  in  the  definition  phase  of  the  project 

•  outside  consulting  costly  and  not  always  appropriate  in  content  or  timing 

•  data  collection  is  not  easy  and  most  of  the  times  not  adequately  representative  to  prove 
the  point  conclusively 

•  getting  people  for  training  is  not  easy  due  to  priority  to  project  work 

4.19.3  Unique  or  Distinguishing  Practices 

Internal  SEPG  workshop  being  held  bi-annually  to  share  best  ideas  and  practices. 

Technology  change  for  the  organization’s  infrastructure  is  well  managed  (Unicenter,  virus 
control,  etc.). 

4.19.4  Peopie  and  Cultural  Issues 

The  selection  of  authors  and  reviewer  has  to  be  done  very  carefully.  Some  of  the  key  criteria 
to  keep  in  mind  are 

•  The  authors  and  reviewers  should  be  respected  members  of  the  organization,  else  exter¬ 
nal  help  should  be  sought. 

•  Selected  members  should  have  kept  sufficient  time  allocated  to  their  assigned  tasks. 

•  Selected  members  should  be  stakeholders,  i.e.,  they  shall  remain  accountable  during  im¬ 
plementation  stage. 

4.19.5  Continuing  improvement 

The  experience  of  existing  SEI  Software  CMM  Level  5  organizations  shows  that  Level  5  ma¬ 
turity  does  not  provide  the  ultimate  performance,  but  it  is  a  critical  milestone  in  their  overall 
journey  of  continuous  process  improvement.  Level  5  maturity  does  provide  a  very  sound  ba¬ 
sis  or  foundation  to  launch  performance  driven  programs  such  as  Six  Sigma  programs.  SISL 
has  already  drawn  up  its  strategy  to  reach  Level  5  maturity  along  with  the  next  step  to  follow. 
This  means  that  closure  of  Software  CMM  program  overlaps  with  the  launch  of  the  next  pro¬ 
gram. 

At  SISL,  the  Personal  Software  Process  Program  is  planned  to  succeed  the  Software  CMM 
Program.  People  CMM  activities  are  also  being  executed  in  parallel.  Organizations  working 
at  higher  levels  of  maturity  may  find  that  processes  are  in  place,  but  they  are  not  as  effective 
as  they  would  like  them  to  be.  In  other  words  the  processes  exist,  but  their  performance  is 
still  not  meeting  the  business  challenge.  This  is  largely  due  to  the  fact  that  the  organizations 
performing  at  Managed  and/or  Optimizing  Level  may  have  put  processes  and  mechanisms  in 
place  but  have  not  created  appropriate  motivations  to  drive  the  performance  of  the  organiza¬ 
tion.  This  may  mean  that  the  organization  has  become  more  bureaucratized  and  is  unable  to 
challenge  the  individuals  adequately.  At  Level  5  maturity,  the  organization  needs  a  systematic 
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flow  of  project  information  more  or  less  in  bureaucratic  manner.  But  software  individuals 
expect  freedom  to  feel  motivated.  How  can  we  solve  this  paradox? 

4.19.6  Summary 

Due  to  the  improvement  activities  SISL  is  able  to  achieve  the  following. 

•  improved  management  of  external  and  internal  commitments 

•  effective  project  management  with  ability  to  “manage  change” 

•  quicker  post-delivery  sign-off  as  requirements  are  better  understood,  leading  to  proper 
implementation 

•  reduced  maintenance  cost  as  result  of  better  documentation 


4.20  Tata  EIxsi  Ltd  (TEL),  Bangalore,  India 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor(s) 

Point  of  Contact 
Web  Page 

Size  of  the  Organization 
Typical  Program  Size 
Primary  Application  Domain(s) 


5 

June  2000 
Pradeep  Udhas 

M  Thangarajan,  mtr@teiI.soft.net 

www.tataelxsi.com 

400 

5K  to  150K  source  lines  of  code 

Networking  protocol  development  (TCP/IP,  ATM,  ISDN,  SNMP, 
BGP,  etc.) 

System  development  (embedded  systems,  ASIC  design,  VHDL  and 
Verilog  modeling,  firmware  including  DSP) 

Visual  computing  (2D/3D  graphics,  animation,  data  visualization, 
etc.) 

Internet  and  intranet  group  (Web  enabling  of  products) 


The  Design  and  Development  Center  of  Tata  EIxsi  Limited,  in  Bangalore,  was  assessed  at 
Software  CMM  Level  5  in  June  2000.  This  achievement  marked  a  significant  milestone  in  the 
organization’s  quest  for  continued  process  maturity  and  improvement. 

The  Quality  Management  System  of  our  organization  was  first  designed  to  meet  the  ISO 
9001,  TicklT  guidelines  and  we  obtained  the  certification  in  February  1997.  Subsequently  our 
processes  were  greatly  enhanced  to  meet  the  requirements  of  the  Capability  Maturity  Model 
and  the  necessary  implementations  were  carried  out  over  the  next  three  years.  The  organiza- 
tion  was  first  assessed  at  Level  4  in  August  1999  and  subsequently  the  organization  was  as¬ 
sessed  at  Level  5  in  June  2000,  using  the  CB A IPI  method.  The  assessment  was  conducted  by 
KPMG  (Lead  Assessor:  Pradeep  Udhas). 
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4.20.1  ROI  and  Improvement  Trend  Data 

A  key  factor  in  our  success  has  been  the  ability  to  show  that  our  efforts  have  added  value  to 
the  overall  organization.  As  a  result  of  widespread  u^e  of  data,  our  productivity  has  almost 
doubled,  estimation  accuracy  has  improved  from  25%  to  12%,  our  outgoing  defect  density 
has  reduced  from  3  defects  /  KLOC  to  0.75  defects  /  KLOC,  and  there  has  been  a  25%  im¬ 
provement  in  the  effort  overrun  from  what  it  was  three  years  ago. 

4.20.2  Barriers  to  Achieving  High  Maturity 

We  still  have  some  way  to  go  to  achieve  our  ultimate  goal  of  continuous  process  improve¬ 
ment  as  a  way  of  life  in  the  organization.  Significant  issues  continue  to  place  challenges  in 
our  path.  These  include: 

•  managing  growth  due  to  high  infusion  of  new  recruits 

•  new  customers  with  unknown  process  maturity  levels 

•  focus  from  projects  to  product  development 

We  have  made  great  progress  over  the  last  three  years  and  we  expect  to  continue. 

4.20.3  Unique  or  Distinguishing  Practices 

The  QMS  is  available  to  all  employees  through  the  intranet.  Past  data  from  projects,  best 
practices,  and  customer  input  are  shared  across  the  organization  through  the  “process  data¬ 
base”  and  “knowledge  management  database”  on  the  intranet.  Organization  baselines  are  es¬ 
tablished  in  line  with  the  organization’s  quality  policy.  Each  project  sets  tailored  quality  ob¬ 
jectives  based  on  the  organization  baselines,  and  stage-wise  analysis  is  done  to  arrive  at 
corrective  actions  if  the  objectives  are  not  met.  Statistical  process  control  methods  such  as 
control  charts,  Rayleigh  curves,  and  Gompertz  curves  are  used  widely  across  projects  to 
measure  the  effectiveness  of  verification  and  validation  activities.  Defect  prevention  activities 
are  carried  out  using  defect  prevention  guidelines  and  orthogonal  defect  classification  (ODC). 
The  quality  function  plan  is  in  line  with  the  business  plan  of  the  organization.  Key  Result 
Area  (KRA)  sheets  are  available  for  SEPQ  SQA  and  training  functions,  and  the  goals  are 
aligned  to  the  business  goals  of  the  organization.  The  SEPG  and  the  SQA  group  are  responsi¬ 
ble  for  process  improvement  initiatives  and  ensure  information  is  disseminated  across  the 
organization. 

4.20.4  People  and  Cultural  Issues 

Extensive  influx  of  new  recruits,  especially  at  the  entry  level,  is  a  great  challenge  that  we  are 
currently  facing.  We  have  initiated  a  three-day,  hands-on  program  for  new  recruits  where  the 
participants  are  exposed  to  all  the  life-cycle  stages  and  the  corresponding  processes.  This 
makes  them  familiar  with  the  process  flow  of  the  organization  prior  to  joining  a  live  project. 
The  last  two  programs  were  highly  rated  and  the  results  have  been  significant. 
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4.20.5  Continuing  Improvement 

In  order  to  meet  the  challenges  of  retaining  employees  and  attracting  fresh  talent,  our  organi- 
zation  has  decided  to  adopt  the  People  CMM  model.  We  believe  that  the  co-existence  of  good 
software  development  practices  using  the  Software  CMM  along  with  good  strategies  to  man¬ 
age  people  with  the  help  of  the  People  CMM  could  help  meet  this  challenge. 


86 


CMU/SEI-2001-SR-014 


4.21  Tata  Consultancy  Services,  India 


Twelve  centers  of  TCS  have  been  assessed  at  Level  4  or  5.  These  follow. 


TCS  Center 

Level 

Lead  Assessor 

Date 

TCS,  SEEPZ,  Mumbai 

4 

Ron  Radice 

TCS,  US  West,  Chennai 

5 

Ron  Radice 

April  1999 

5 

Ron  Radice 

July  1999 

5 

Ron  Radice 

August  1999 

TCS,  Shollingnallur,  Chennai 

5 

Ron  Radice 

TCS,  Calcutta 

5 

Ron  Radice 

TCS,  Bangalore 

5 

Ron  Radice 

January  2000 

TCS,  Lucknow 

5 

Radhika  Sokhi 

Jack  Harding 

January  2000 

TCS,  Ahmedabad 

4 

Ron  Radice 

March  2000 

TCS,  Hyderabad 

5 

Gargi  Keeni 

Jack  Harding 

May  2000 

TCS,  Ambattur,  Chennai 

5 

Ron  Radice 

July  2000 

5 

Jack  Harding 

July  2000 

TCS,  Ahmedabad 

5 

P.  Suresh 

Ron  Radice 

November  2000 

TCS,  Gurgaon  II,  New  Delhi 

5 

Ron  Radice 

February  2001 

Point  of  Contact 

Gargi  Keeni,  gkeeni@mumbai.tcs.co.in 

Web  Page 

www.tcs.com 

Size  of  the  Organi¬ 
zation 

9,000  in  12  TCS  centers.  The  staff  strength  of  the  individual  centers  range  from 

150  to  1600. 

Typical  Program 

Size 

Number  of  people  per  typical  project:  10-30 

Number  of  lines  of  code  per  typical  project:  10  KLOC-100  KLOC 

Number  of  function  points  per  typical  project:  400  FP-4000  FP 

Primary  Application 
Domain(s) 

Examples  of  the  various  types  of  projects  executed: 
development 
conversion 
maintenance 
package  implementation 
engineering  services 
product  data  management 
architecture  and  technology  consulting 
language  processing 
strategic  consulting 
quality  consulting 
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In  1992-93,  TCS  defined  and  documented  its  Quality  Management  System  (QMS).  All  the 
TCS  Centers  were  ISO  certified  in  the  period  from  1993  to  1998.  In  1996,  TCS  decided  to 
enhance  the  QMS  to  align  it  with  the  Software  CMM  Version  1.1.  At  present  12  of  the  17 
development  centers  have  been  assessed  to  be  at  Level  5.  Six  Sigma  is  being  implemented  in 
two  centers  of  TCS.  Currently,  TCS  is  in  the  process  of  benchmarking  its  people  practices 
against  People  CMM  Version  1.0. 


Growth  in  Human  Resource 


Figure  31:  TCS  Growth  in  Human  Resources 

4.21.1  ROI  and  Improvement  Trend  Data 


Development  Projects  -  Trend  of  effort  slippages 
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Figure  32:  TCS  Trend  of  Percentage  Effort  Slippage  in  Development 
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Percentage  occurrence 
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Figure  35:  TCS  Trend  in  Schedule  Slippage  in  Maintenance 

The  figure  below  shows  the  trend  in  the  capability  for  on-time  delivery  of  one  of  the  TCS 
centers  implementing  Six  Sigma.  The  capability  for  on-time  delivery  has  improved  from 
2.85a  to  4.63a.  These  improvements  have  been  possible  through  a  number  of  defect  preven¬ 
tion  and  productivity  improvements  implemented  by  the  Six  Sigma  projects  in  the  center. 


Process  Capability  Analysis  for  Sch.slippage 

LSL  USL 


Figure  36:  Process  Capability  Analysis  for  TCS  Schedule  Slippage 

4.21.2  Barriers  to  Achieving  High  Maturity 

There  were  a  number  of  challenges  in  deploying  the  key  practices  of  Level  4  and  Level  5. 
Some  of  these  have  been  listed  below.  In  certain  instances,  the  remedial  method  adopted  is 
also  mentioned. 


90 


CMU/SEI-2001-SR-014 


•  creating  confidence  in  people  that  quality  pays  in  the  long  run 

•  resistance  towards  process  change  in  on-going  projects 

•  prioritizing  process  change 

•  continuous  training  of  people  on  new  initiatives  /  processes,  applying  standard  processes 
and  institutionalization  of  these  processes  considering  the  size  of  the  organization,  and 
high  mobility  of  personnel  (persons  going  on-site,  returning  from  overseas,  new  join¬ 
ers,  people  turnover,  etc.).  This  was  further  compounded  due  to  the  wide  variety  of  pro¬ 
jects  executed  across  the  centers. 


Metrics-related  challenges  remain: 

•  understanding  SPC  and  convincing  project  teams  that  it  works  for  software 

•  application  of  statistical  techniques  to  ensure  Quantitative  Process  Management  (QPM), 
Software  Quality  Management  (SQM)  and  Defect  Prevention  (DP) 

•  logging  of  time  sheets 

•  calculating  ROI  for  benefits 

•  for  online  analysis  of  metrics,  faster  aggregation  of  data  was  required.  In-house  tools 
provided  the  support  for  aggregation  of  data  and  drawing  SPC  charts. 

•  motivating  people  to  log  defects 

•  attribution  of  appropriate  causes  for  defect  prevention  Rigorous  training/facilitation  ses¬ 
sions  required  educating  practitioners  in  how  to  assign  appropriate  causes  and  avoid  ones 
like  “Others.” 

•  Submission  of  monthly  status  reports  by  all  the  project  leaders  to  SEPG  on  time  was  not 
regular.  Used  Integrated  Project  Management  System  (IPMS)  tool,  developed  by  TCS,  to 
assist  the  Project  Leaders  to  log  effort  for  different  key  process  areas. 

•  initial  participation  of  DP,  PCM,  and  TCM  project  primes  in  the  core  group  meeting  was 
very  thin.  Participation  of  staff  for  raising  process  improvement  proposals  (PIPs)  was  ini¬ 
tially  thin  too. 

•  implementing  processes  in  short  duration  projects.  Appropriate  tailoring  guidelines  were 
prepared  to  tackle  this  issue. 

•  keeping  Integrated  Project  Management  System  (IPMS)  up-to-date  with  the  changing 
processes 

•  manage  projects  that  are  being  executed  at  customer  locations.  Standardized  tools  used 
which  could  be  used  at  customer  site  and  merged  at  periodic  intervals  into  the  centralized 
database. 

•  tracking  PIPs  manually  was  difficult  leading  to  delay  in  feedback  and  implementation. 
Tool-based  PIP  management  increased  visibility  into  the  system. 


4.21.3  Unique  or  Distinguishing  Practices 

•  Encouraging  consultants  to  get  CQA  Certification.  The  target  is  to  have  at  least  10%  of 
the  total  strength  as  certified  CQAs. 


CMU/SEI-2001-SR-014 


91 


•  Encouraging  /  sponsoring  employees  to  become  members  of  professional  bodies  (IEEE. 
ACM,  etc.)  in  order  to  foster  learning.  (There  are  2500  IEEE  members  currently.) 

•  Audit  Pool,  Reviewers’  Pool  and  Assessors’  Pool  comprises  of  experts  from  the  project 
teams. 

•  Quality  Assurance  Group  (QAG)  is  entrusted  with  the  responsibility  of  facilitating  the 
projects  in  deploying  the  Quality  Management  System  (QMS).  The  Audit  group  is  an  in¬ 
dependent  group,  which  is  responsible  for  verification  of  process  and  product  compli¬ 
ance. 

•  Usage  of  lessons  learned  and  best  practices  shared  within  and  across  projects  as  well  as 
centers  of  TCS  through  the  use  of  the  knowledge  management  tool 

•  Periodic  QAG  Audit  Group,  and  SEPG  meetings  are  organized.  These  forums  also  pro¬ 
vide  knowledge  sharing  opportunities. 

•  Higher  reliance  on  SPC  to  quantitatively  manage  the  product  and  process  quality. 

•  Usage  of  the  project  management  tool  IPMS  across  all  projects,  which  serves  as  the  sin¬ 
gle  repository  for  process  data. 

•  Project  start-up  meetings  improve  intergroup  commitments  and  team  building. 

•  Creation  of  roles  such  as  process  owners  for  DP,  PCM  and  TCM  at  the  organization  level 
and  distinct  project  primes  for  DP,  PCM,  and  TCM  at  project  level. 

•  Service  level  agreements  (SLAs)  defined  for  all  support  groups  who  measure  their  per¬ 
formance  against  the  SLAs  and  constantly  strive  to  refine  them. 

•  Specialized  training  courses  on  soft  skills  like  team  building.  Six  Thinking  Hats,  Lateral 
Thinking,  negotiations,  and  customer  management. 

•  Using  causal  analysis  for  product  defects,  process  defects,  customer  complaints,  and  cus¬ 
tomer  satisfaction  feedback  and  proposal  won  /  lost  analysis. 

•  Team  members  given  the  responsibility  to  present  the  project  information  and  status  dur¬ 
ing  project  management  reviews 

•  Training  of  20  days  per  year  is  mandated  for  every  person  in  the  center,  which  aims  at 
orienting  the  individual  towards  continuous  learning. 

•  Organized  structure  for  Six  Sigma  culture  at  some  TCS  centers. 

•  Six  Sigma  projects  for  DP,  QPM,  SQM,  and  productivity  improvement. 


4.21.4  People  and  Cultural  Issues 

The  SPI  initiative  has  brought  a  strong  cultural  change  among  people  of  the  organization.  The 
mindset  of  the  people  has  changed  from  reactive  to  proactive  management. 

Expectations  of  management  and  staff  in  terms  of  process  effectiveness  is  high  and  pace  of 
change  for  process  improvement  is  expected  to  be  high.  A  large  number  of  people  are  willing 
to  contribute  and  expectations  need  to  be  handled  well. 
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Customer  expectations  too  are  high,  and  it  is  not  always  understood  that  high  maturity  of  the 
subcontractor  is  not  a  silver  bullet  and  still  requires  maturity  of  customer  processes  in  order 
that  mutual  objectives  be  met.  Some  customers  have  mixed  feelings  and  expect  that  high  ma¬ 
turity  processes  are  an  overhead  on  cycle  time. 

Employee  morale  has  improved  due  to  a  formal  mentor-mentee  program.  This  has  been  insti¬ 
tutionalized  for  sometime.  Each  person  who  Joins  the  organization  is  associated  with  a  men¬ 
tor,  who  helps  the  person  with  their  personal  problems,  career,  and  technical  growth  in  the 
organization. 

Skills  building  has  been  the  cornerstone  for  improvement  activities.  Skills  of  individuals  are 
managed  using  the  Skills  Management  System  (SMS)  tool  developed  in-house.  A  role  profile 
is  defined  for  each  role  in  TCS.  Skills  of  all  staff  are  logged  into  this  tool  and  based  on 
his/her  chosen  career  path,  career  development  plans  are  prepared  and  training  needs  are 
identified.  Individual  skills  are  accordingly  build. 

There  is  an  extensive  90-day  Initial  Training  Program  (TTP),  which  all  fresh  joinees  under¬ 
take.  This  induction  training  covers  most  aspects  of  software  engineering,  process  method¬ 
ologies,  and  soft  skills. 

Specific  training  programs  are  given  in  the  areas  in  which  team  members  are  required  to 
work.  Post-evaluation  feedback  helps  to  improve  the  training  program. 

Specialized  training  courses  on  soft  skills  such  as  Team  Building,  Six  Thinking  Hats,  Lateral 
Thinking,  Negotiations,  Customer  Management,  Working  in  Teams  Effectively,  etc.,  help 
the  project  leaders  to  perform  their  tasks  effectively.  TCS  has  a  variety  of  such  courses  and 
all  persons  eligible  to  become  project  leaders  undergo  these  training  programs. 

With  the  organization  operating  at  higher  maturity  levels,  it  becomes  a  continuous  challenge 
to  train  the  fresh  joinees  on  SPI.  In  TCS,  this  is  handled  through  the  Initial  Training  Program 
(TTP).  Apart  from  turnover,  people  in  TCS  keep  going  on  overseas  assignments  for  long  dura¬ 
tion.  The  Quality  Groups  face  a  continuous  challenge  of  keeping  these  people  up-to-date  with 
the  SPI  activities  that  would  have  happened  in  their  absence. 

4.21.5  Continuing  Improvement 

The  culture  strongly  supports  quality  and  continuous  improvement  within  the  organization. 

To  sustain  the  pace  of  continuous  improvement,  the  quality  structure  is  well  maintained  with 
more  emphasis  towards  Defect  Prevention,  Technology,  and  Process  Change.  In  each  of  these 
areas,  apart  from  organization  level  process  owners,  there  exist  process  primes  at  the  project 
level.  Regular  evaluation  of  the  improvement  proposals  is  done  and  necessary  improvement 
measures  are  adopted. 
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The  improvement  objectives  are  aligned  to  the  business  goals  of  the  center.  Centers  are  al¬ 
ways  on  the  look  out  for  standards  and  methodologies  that  help  in  identifying  new  areas  and 
sustaining  continuous  improvement.  Some  of  these  new  initiatives  are: 

•  Six  Sigma:  For  example,  one  center  has  a  target  to  complete  100  Six  Sigma  projects  for 
2001,  which  includes  projects  on  design  for  Six  Sigma  focussing  on  commercial  quality 
and  product  quality 

•  sustain  the  participation  of  all  in  suggestion  schemes  /  PIPs 

•  causal  analysis  of  customer  complaints  and  customer  dissatisfaction  parameters 

-  causal  analysis  of  won  /  lost  proposal 

-  participating  in  various  forums  such  as  meetings,  seminars,  presenting  and  publishing 
papers,  etc. 

-  continuously  trying  to  improve  upon  processes  and  metrics  for  short  duration  projects 

-  improve  tools  for  SPI 

-  improving  requirements  capture  in  non-English  speaking  countries 

-  the  estimation  procedures  are  being  strengthened  to  cope  with  the  rapid  technology 
changes 

-  strengthening  our  project  management  processes 

•  TCS  plans  to  go  in  for  reassessment  for  each  of  its  centers  that  have  achieved  Level  5 
every  year  using  Software  CMM  v  1.1  to  re-affirm  the  organization’s  continuously  im¬ 
proving  policy. 


TATA  Business  Excellence  Model  (TBEM)  (on  the  lines  of  the  Malcolm  Baldrige  National 
Quality  Award  -  MBNQA)  is  the  overall  business  excellence  model  adopted  by  TCS.  In  this 
context,  apart  from  ISO  9001 : 1994  and  Software  CMM,  TCS  is  working  on: 

•  benchmarking  with  People  CMM  Version  1.0 

•  transitioning  to  CMMI 

4.21.6  Summary 

Level  5  has  given  a  structured  process  framework,  which  has  facilitated  success  of  initiatives 
such  as  Six  Sigma. 

Level  5  has  given  market-place  competitiveness. 

Level  5  has  streamlined  the  internal  processes  such  as  metrics,  tools  development  and  de¬ 
ployment,  data-based  decisions  which  help  increase  the  empowerment  at  all  levels. 

Being  at  Level  5  means  a  lot  of  responsibility  as  the  company  sets  trends  in  the  industry  by 
sharing  its  best  practices.  It  continuously  improves  processes  and  therefore  needs  a  strong 
mechanism  to  sustain  the  momentum.  With  changing  business  models  and  technology  it  is  a 
tough  challenge  to  keep  the  movement  on. 
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4.22  U.S.  Army  Aviation  and  Missile  Command 
(AMCOM)  Software  Engineering  Directorate 
(SED),  Redstone  Arsenal,  AL 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor(s) 

Point  of  Contact 
Web  Page 

Size  of  the  Organization 
Typical  Program  Size 


Primary  Application  Domain(s) 


4 

April  2000 
Dave  Zubrow 

Jackie  Langhout,  Jackie.Langhout@sed.redstone.army.mil 
http :// w  w  w  .redstone .  army .  mi  1/amrdec/sed/ 

150  government  employees  and  600  contractors 
5-10  people  per  typical  project 
lOK  SLOC  for  new  development  projects 
340K  SLOC  maintained  for  sustainment  projects 
Military  software 

Applications  include:  trainers,  simulators,  emulators,  test  devices  / 
tools,  and  tactical  systems 


As  the  Army  Aviation  and  Missile  Command’s  (AMCOM’s)  Life  Cycle  Software  Engineer¬ 
ing  Center,  the  SED’s  mission  is  to  provide  mission  critical  computer  resource  expertise  to 
support  AMCOM’s  weapon  systems  over  their  life  cycle.  This  support  includes  providing 
software  expertise  to  AMCOM  weapon  systems,  providing  affordable  and  effective  post  de¬ 
ployment  software  support,  providing  air  defense  interoperability  engineering  and  testing  and 
applying  the  latest  in  software  technologies  to  these  efforts. 

Of  the  750  personnel  at  the  AMCOM  SED,  approximately  225  personnel  are  involved  in  the 
development  /  sustainment  of  software.  The  remaining  technical  personnel  are  involved  in 
such  tasks  as  supporting  Army  weapon  system  managers  in  the  development  of  software  ac¬ 
quisition  plans  and  standards,  performing  evaluations  of  prime  contractor’s  software  devel¬ 
opment  activity,  and  providing  interoperability  engineering  support. 

The  AMCOM  SED  utilizes  the  skills  of  both  government  and  contractor  employees  with  a 
team  approach  to  provide  the  highest  quality  services  and  products  to  its  customers.  It  does 
not  use  subcontracting  as  a  means  for  developing  or  maintaining  software.  The  AMCOM 
SED  staffs  each  project  with  a  mix  of  government  and  contractor  personnel  who  work  to¬ 
gether  as  an  integrated  team  using  the  same  set  of  project  standards  and  procedures. 

The  AMCOM  SED  initiated  its  software  process  improvement  efforts  in  1991.  The  organiza¬ 
tion  established  the  Software  Engineering  Institute’s  (SETs)  Software  Capability  Maturity 
Model  (Software  CMM)  as  the  framework  to  be  utilized  in  developing  and  maturing  the 
organization’s  software  process.  The  AMCOM  SED  conducted  a  self-assessment  in 
September  1991  and  was  rated  Level  1.  A  CBA IPI  was  performed  in  May  1994  and  resulted 
in  a  Level  2  rating.  The  AMCOM  SED  conducted  a  third  assessment  using  the  CBA  EPI 
method  in  November  1996.  That  assessment  concluded  that  the  SED  was  performing  at  the 

Level  3  maturity.  In  April  2OQ0rthcAMG0M-SED  conducted  another-assessmcnt  using  the 
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ity.  In  April  2000,  the  AMCOM  SED  conducted  another  assessment  using  the  CBA IPI 
method  and  achieved  a  Level  4  rating. 

4.22.1  ROI  and  Improvement  Trend  Data 

As  stated  above,  the  AMCOM  SED  initiated  its  software  process  improvement  efforts  in 
1991  and  has  maintained  a  consistent  focus  throughout  the  years.  The  estimation  and  plan¬ 
ning  activities  have  been  greatly  improved  thus  decreasing  the  opportunity  for  project  fail¬ 
ures.  The  organization’s  productivity  (total  SLOC  developed  /  effort  expended)  has  doubled 
since  initiation  of  the  improvement  program.  In  addition  to  an  increase  in  productivity,  the 
software  improvement  program  has  had  a  positive  impact  on  the  growth  in  the  amount  of 
work  received  from  customers.  Since  1992  the  workload  has  increased  by  a  factor  of  four. 
The  AMCOM  SED’s  products  continue  to  be  of  high  quality,  as  the  defect  density  in  the  de¬ 
livered  products  average  less  than  one  per  KSLOC.  The  cost  benefit  of  peer  reviews  has  been 
analyzed  and  it  was  determined  that  finding  defects  through  peer  reviews  was  nine  times 
more  cost  effective  than  finding  defects  through  testing. 

4.22.2  Barriers  to  Achieving  High  Maturity 

Understanding  the  Level  4  key  practices  was  a  barrier  for  the  AMCOM  SED.  The  terminol¬ 
ogy  in  Levels  4  and  5  of  the  Software  CMM  was  difficult  to  grasp.  Because  there  were  so 
few  organizations  that  had  reached  Level  4  when  the  AMCOM  SED  journey  began,  lessons 
learned  were  not  readily  available. 

The  AMCOM  SED’s  metrics  program  had  to  be  significantly  revised  to  support  the  Level  4 
activities.  The  metrics  program  was  modified  to  measure  process  performance  and  quality 
throughout  the  project  life  cycle. 

4.22.3  Unique  or  Distinguishing  Practices 

The  AMCOM  SED’s  analysis  of  metrics  data  has  allowed  for  a  better  understanding  of  our 
process.  The  analysis  of  peer  review  data  provides  projects  with  timely  insight  into  product 
quality  issues.  The  use  of  automation  and  web-based  tools  has  significantly  enhanced  the 
AMCOM  SED’s  software  process  improvement  program. 

4.22.4  People  and  Cultural  Issues 

The  employees  at  the  AMCOM  SED  have  great  pride  in  their  software  development  process. 
Employee  turnover  is  not  an  issue.  The  organizational  training  program  reinforces  the  process 
improvements  that  are  initiated.  The  organization  has  experienced  remarkable  growth  in  the 
past  10  years  and  the  software  process  improvement  program  has  supported  that  growth. 

Each  project  has  an  assigned  mentor  from  the  Software  Engineering  Process  Group,  which 
supports  the  project’s  implementation  of  the  defined  software  process  and  measurement  pro¬ 
gram. 
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4.22.5  Continuing  Improvement 

The  current  and  future  activities  for  the  AMCOM  SED  are  ( 1)  strengthen  the  software  devel¬ 
opment  process  based  upon  the  recommendations  from  the  recent  assessment;  (2)  migrate  to 
the  CMMI,  thus  expanding  process  improvement  into  other  areas  in  our  organization:  (3)  pi¬ 
lot  PSP/TSP  on  1-2  software  projects. 


4.22.6  Summary 

The  AMCOM  SED  has  received  much  recognition  because  of  the  Level  4  rating.  Organiza¬ 
tions  throughout  the  community  frequently  seek  support  from  AMCOM  SED  when  embark¬ 
ing  upon  a  software  process  improvement  initiative.  Customers  view  the  AMCOM  SED  as  a 
center  of  excellence  and  have  broadened  the  areas  of  support  that  AMCOM  SED  provides. 
The  organization’s  defined  software  process  is  remarkably  more  entrenched  into  the 
AMCOM  SED  culture  because  of  the  efforts  to  continue  up  the  maturity  scale. 


4.23  Software  Engineering  Division  of  the  Ogden  Air 
Logistics  Center  (00-ALC/TIS),  Ogden,  UT 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor(s) 

Point  of  Contact 
Web  Page 

Size  of  the  Organization 
Typical  Program  Size 
Primary  Application  Domain(s) 


Level  5 
July  1998 
Brian  Larman 

Jim  Vanfleet,  Jim.Vanfleet@Hill.af.mil 
http://www,tis.hill.af.mil 

TIS  employs  approximately  560  engineers,  technicians,  configura¬ 
tion  management  specialists,  and  support  personnel 

Software  project  size  can  vary  from  a  one  engineer  project  to  larger 
development  projects  of  75  or  more  engineers 

Automatic  Test  Equipment  (ATE)  software,  Operational  Flight  Pro¬ 
grams  (OFPs),  and  Mission  Planning  software  for  F-16,  A-10,  B-52, 
B-1 ,  Minute  Man,  and  Peacekeeper  programs  /  systems 


The  Software  Engineering  Division  at  Hill  AFB  (TIS)  initiated  its  software  process  im¬ 
provement  effort  in  1991  and  three  years  later  was  assessed  as  a  Software  CMM  Level  2  or¬ 
ganization.  TIS  was  reassessed  in  1995  at  Software  CMM  Level  3.  In  July  of  1998,  TIS  be¬ 
came  the  first  government  organization  to  be  assessed  as  a  Software  CMM  Level  5. 


In  addition  to  the  Software  CMM,  TIS  also  utilizes  the  Personal  Software  Process  (PSP)  and 
the  Team  Software  Project  (TSP).  TIS  has  three  projects  that  utilize  the  Team  Software  Pro¬ 
ject  (TSP),  comprised  of  30+  Personal  Software  Process  (PSP)  trained  engineers.  There  are 
four  PSP  certified  instructors  and  two  TSP  launch  coaches  in  the  division. 
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ROI  and  Improvement  Trend  Data 

A  discussion  of  the  benefits  that  TIS  has  received  would  be  incomplete  without  addressing 
some  of  the  problems  that  have  been  encountered  in  trying  to  calculate  ROI.  At  the  beginning 
of  the  TIS  SPI  effort,  the  organizational  capabilities  were  not  known.  As  a  result,  the  initial 
assumptions  /  estimates  were  difficult  to  make  and  are  easy  to  challenge. 

Because  TIS  is  a  non-profit  organization,  cost  avoidance  or  savings  was  used  rather  than  in¬ 
creased  profit.  The  ROI  formula  that  was  used  was:  ROI  =  Savings  /  Investment.  Investment 
is  easy  to  capture,  however,  determining  what  constitutes  savings  is  difficult. 

The  ROI  calculations  for  TIS  were  based  on  a  ten-year  moving  window.  ROI  ranged  from  4: 1 
to  19: 1,  depending  on  the  project.  Level-of-effort  projects  yielded  the  highest  ROI. 

Some  types  of  improvement  are  more  tangible  than  others.  Some  of  the  cost  reduction  data  is 
as  follows:  34%  reduction  per  source  line  of  code  (SLOC),  39%  decrease  in  regression  test¬ 
ing,  and  65%  to  75%  reduction  in  cycle  time  for  MIPs  and  OCPs.  Schedule  improvements, 
quality  improvements,  improved  morale  and  customer  satisfaction  are  impossible  to  quantify 
in  terms  of  dollar  savings,  but  these  benefits  have  been  observed  and  are  attributed  to  TIS’s 
SPI  efforts. 

4.23.2  Barriers  to  Achieving  High  Maturity 

Some  barriers  that  have  been  overcome  in  achieving  Level  5  are:  diverse  projects/workloads, 
projects  that  had  a  significant  amount  of  hardware  development,  and  difficulty  understanding 
the  Level  4  and  5  issues. 

TIS’s  size  and  diverse  workloads  caused  some  unique  problems  in  achieving  high  maturity. 
TIS  utilized  the  product  line  approach  to  overcome  these  barriers.  Diverse  projects  (Auto¬ 
matic  Test  Equipment  [ATE],  Operational  Flight  Programs  [OFP]  /  Mission  Planning,  and 
projects  with  a  large  percentage  of  hardware  development)  were  organized  into  product  lines. 
The  product  lines  share  similar  processes.  Extended  SEPGs  or  ESEPGs  were  created  to  per¬ 
form  product-line  SPI  activities. 

Another  challenge  TIS  faced  was  understanding  and  implementing  the  Level  4  and  5  key 
process  areas.  In  1995,  when  TIS  was  assessed  at  Level  3,  there  were  a  limited  number  of 
high  maturity  organizations.  Information  and  resources  regarding  the  implementation  of  high 
maturity  key  process  areas  was  nonexistent.  TIS  was  forced  to  “break  new  ground”  to 
achieve  its  Level  5  rating. 

4.23.3  Unique  or  Distinguishing  Practices 

Two  distinguishing  characteristics  of  TIS  are  the  quality  assurance  program  and  the  extensive 
use  of  process  tools  within  the  division. 
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The  Quality  Engineering  Support  Team  (QuEST)  was  developed  to  fill  the  quality  assurance 
role  within  TIS.  QuEST  is  unique  because  their  activities  focus  on  the  processes  used  to  pro¬ 
duce  the  product  rather  than  the  product  itself.  QuEST  regularly  audits  senior  management, 
project  management,  practitioners,  CM,  and  the  SEPG  to  ensure  that  organizational  policies 
and  procedures  are  being  followed.  Findings  are  documented  in  reports  and  action  plans  are 
required  to  address  the  findings.  QuEST  is  an  invaluable  resource  in  furthering  the  SPI  efforts 
within  TIS. 

Tools  have  also  become  an  important  part  of  life  in  TIS.  Due  to  the  large  amount  data  that  our 
processes  and  projects  collect  and  analyze,  development  of  tools  became  necessary  to  man¬ 
age,  analyze,  and  present  the  data.  Configuration  management  tools  and  process  tools  are  also 
used  extensively.  Most  of  these  tools  have  been  developed  in-house,  but  a  few  of  them  are 
commercial  off-the-shelf  products. 

4.23.4  People  and  Cultural  Issues 

Although  losing  key  project  personnel  can  still  be  a  problem,  TIS  has  become  less  reliant  on 
project  “heroes.”  The  time  required  for  an  employee  to  become  knowledgeable  and  produc¬ 
tive  on  a  project  process  has  decreased.  Employees  can  be  moved  to  new  projects  within  the 
organization  with  greater  ease  and  less  risk.  New  employees  are  assigned  mentors  and  re¬ 
ceive  training  on  the  project’s  process  and  procedures.  In  addition,  all  employees  are  required 
to  receive  formal  organization  policy  /  process  training  and  product-line  process  training. 

TIS  has  become  increasingly  dependent  on  process  “heroes”.  Project  managers  who  are 
knowledgeable  in  the  Software  CMM  and  TIS  policy  are  invaluable.  One  thing  that  has 
helped  TIS  to  remedy  this  problem  is  to  rotate  perspective  managers  and  leads  through  SEPG 
and  QuEST.  An  understanding  of  the  Software  CMM  and  TIS  policy  is  gained.  Working  in 
these  groups  allows  the  individual  to  be  more  competitive  for  leadership  positions  in  the  divi¬ 
sion. 


4.23.5  Continuing  Improvement 

The  Software  Engineering  Division  is  currently  applying  process  improvement  to  our  current 
Technology  Change  Management  (TCM)  process.  Although  benefit  has  been  derived  from 
our  current  TCM  process,  changes  are  needed.  A  new  database  and  automated  process  are 
currently  being  developed  to  perform  the  TCM  process. 

TIS  has  the  challenge  of  bringing  in  new  and  diverse  workloads.  Part  of  the  challenge  is  deal¬ 
ing  with  customers  who  are  Level  1  (e.g.,  don’t  want  to  document  or  establish  requirements). 
Some  of  these  workloads  are  very  different  from  the  work  TIS  has  traditionally  performed. 
Some  projects  have  a  large  amount  of  hardware  development  with  very  little  software  devel¬ 
opment.  TIS  has  struggled  to  develop  new  project  processes  tailored  from  the  organization’s 
Standard  Engineering  Process  (SEP). 
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4.23.6  Summary 

Because  of  TIS’s  SPI  efforts,  the  division  has  realized  improvements  in  cost,  schedule,  and 
quality.  In  addition,  achieving  a  Level  5  rating  has  also  had  a  positive  effect  on  TIS’s  reputa¬ 
tion  and  image  in  the  DOD  and  SPI  community.  Improvements  to  efficiency,  product  quality, 
and  image  have  improved  relationships  with  TIS’s  customers  and  have  helped  to  attract  new 
workload. 


4.24  TELOS*OK,  Lawton,  OK  73501 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor(s) 

Point  of  Contact 
Web  Page 

Size  of  the  Organization 
Typical  Program  Size 
Primary  Application  Domain(s) 


4 

November  1997 
Don  Couch 

Phil  Sperling,  sperlips@fssec.army.mil 

www.telosok.com 

270 

18  people,  1.2  million  LOC 

Military  systems,  real-time,  and  embedded 


4.24.1  ROI  and  Improvement  Trend  Data 

The  investment  has  been  approximately  600K  dollars  per  year,  since  1990.  The  amount  of 
code  being  developed  /  maintained  has  increased  by  253%,  without  any  increase  in  staffing. 
The  defect  rate  during  final  /  formal  testing  has  decreased  by  69%.  The  primary  emphasis  of 
the  improvement  program  has  been  to  decrease  defects  and  increase  efficiency  of  production 
and  testing. 


4.24.2  Barriers  to  Achieving  High  Maturity 

In  order  to  achieve  Level  4,  the  primary  barrier  was  education  on  the  Software  CMM  meth¬ 
odology.  Most  of  the  concepts  presented  were  familiar  to  the  organization,  however,  we  had 
to  understand  “CMM-ese.”  Of  course,  there  were  changes  made  in  the  declaration  of  outputs 
for  process  performance  and  product  quality,  however,  these  were  relatively  easy  to  convert. 

4.24.3  Unique  or  Distinguishing  Practices 

We  have  a  strong  infrastructure  that  supports  the  monitoring  of  process  compliance,  which  is 
coupled  with  a  relatively  easy  mechanism  to  enhance  /  improve  /  change  the  process.  This 
infrastructure  is  supported  by  quantitative  control  mechanisms  and  an  excellent  reporting 
scheme. 
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4.24.4  People  and  Cultural  Issues 

Not  being  located  in  a  technological  metropolis  creates  a  challenge  for  hiring  and  retaining 
qualified  personnel.  Our  current  staff  is  excellent,  however,  the  competition  is  steep.  This 
really  makes  the  institutionalization  of  a  standard  process  critical. 


4.24.5  Continuing  Improvement 

Our  current  direction  is  the  institutionalization  of  a  program  that  supports  realistic  and  verifi¬ 
able  causal  analysis.  Most  of  our  problems  surround  suppliers.  We  do  a  lot  of  integration 
work,  which  makes  it  critical  to  understand  the  design  of  received  systems.  These  designs 
from  suppliers  are  not  always  complete  and  accurate.  We  plan  to  re-appraise  using  the 
CMMI. 

4.24.6  Summary 

Being  a  Level  4  means  that  our  organization  has  adequate  visibility  into  and  control  of  our 
processes  and  products.  The  organization  (people)  are  excited  and  responsive  to  a  formallized 
process  improvement  program  because  of  the  concrete  returns  they  have  seen  over  the  last  10 
years.  Benefits  are  streamlined  processes,  ease  of  technology  innovation,  and  less  turmoil 
during  testing. 


4.25  U.S.  Air  Force,  Tinker  Air  Force  Base,  OK 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor(s) 

Point  of  Contact 
Size  of  the  Organization 
Typical  Program  Size 
Primary  Application  Domain(s) 


4 

November  1996 
Judah  Mogilensky 

Kelley  Butler,  kelley.butler@tinker.af.mil 
325 

10-60  people  per  project  Varies. 

Department  of  Defense  software,  primarily  Air  Force 

Avionics  (80%)  and  Jet  Engine  (9%)  Test  Program  Set,  Industrial 
Automation  (5%),  and  Jet  Engine  Trending  Software  (5%) 


Registered  to  ISO  9001/Tickrr  since  November  1998.  Maturity  level  history:  SEI  Level  1  - 
1990,  SEI  Software  CMM  Level  2  -  1993;  SEI  Software  CMM  Level  4  -  1996. 


In  1999,  we  were  awarded  the  IEEE  Award  for  Software  Process  Achievement.  The  SEI 
Technical  Report,  Software  Process  Achievement  at  Tinker  Air  Force  Base,  Oklahoma, 
CMU/SEI-2000-TR-014,  September  2000,  that  was  written  in  conjunction  with  winning  the 
award  may  be  found  at 


http://www.sei.cmu.edu/publications/documents/00.reports/00tr014.html 
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Much  of  the  information  requested  for  this  paper  is  in  that  report  and  will  not  be  repeated 
here. 


4.25.1  ROi  and  Improvement  Trend  Data 

See  the  SEI  Technical  Report  [Butler  00]. 

4.25.2  Barriers  to  Achieving  High  Maturity 

Below  is  a  chart  we  prepared  for  the  SEI  technical  report.  We  think  it  shows  that  organiza¬ 
tions  have  to  realize  that  not  all  improvements  will  be  lasting  and  that  they  will  have  to  make 
way  for  other  improvements  at  higher  levels.  The  biggest  barrier  that  we  see  to  getting  to 
Level  4  or  5  is  that  improvement  must  move  beyond  checking  boxes  and  into  the  culture.  A 
Level  4  or  5  organization  truly  believes,  at  all  levels  of  the  organization,  in  what  they  are  do¬ 
ing. 


Table  3:  Tinker  SPI  Time  Line 


Timeframe 

#  Improvements 
Implemented 

#  Still  in  Place  in  1999 

Percent 

1990-1993,  Level  2  in  1993 

45 

11 

24% 

1993-1996,  Level  4  in  1996 

31 

24 

77% 

1996-Present 

22 

22 

100% 

4.25.3  Unique  or  Distinguishing  Practices 

We  don’t  feel  that  this  is  unique  to  our  organization,  but  it  is  unique  to  high  maturity  organi¬ 
zations,  and  that  is  emphasis  on  all  aspects  of  the  process  as  well  as  on  suppliers  and  custom¬ 
ers.  High  maturity  organizations  see  the  big  picture  and  everything  that  is  necessary  to  be 
successful,  and  they  do  whatever  it  takes  to  ensure  success.  They  also  realize  that  they  can 
continually  improve  and  push  the  boundaries,  whether  it  be  in  quality,  productivity,  cost,  or 
schedule. 

4.25.4  People  and  Cultural  Issues 

Unfortunately  in  today’s  economy  recruiting  /  retention  is  a  difficult  issue  but  we  do  find  that 
people  appreciate  working  at  a  higher  maturity  organization,  so  much  so,  that  we  have  had 
several  cases  of  personnel  leaving  and  returning,  or  wanting  to  return,  due  to  the  work  envi¬ 
ronment  and  focus  on  process. 
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4.25.5  Continuing  Improvement 

Our  major  focus  at  this  time  is  SPC.  Additionally  we  are  moving  our  focus  to  the  CMMI- 
SE/SW  and  ISO  9001:2000. 


4.25.6  Summary 

Being  “Level  4”  has  helped  out  in  innumerable  areas.  From  how  others  perceive  the  organi¬ 
zation,  to  organizational  pride,  to  the  bottom  line,  to  making  it  a  better  place  to  work.  It  has 
made  us  a  stronger,  more  aggressive  organization. 


4.26  Wipro  GE  Medical  Systems  Ltd,  Bangalore, India 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor(s) 

Point  of  Contact 
Web  Page 

Size  of  the  Organization 
Typical  Program  Size 
Primary  Application  Domain(s) 


5 

January  1999 
C  Rama  Rao 
Richard  Knudson 

K.Puhazhendi,  k.puhazhendi  @ geind.ge.com 
www.wipro-ge.com 
180 

6  to  10 

Medical  -  diagnostic  imaging 


4.26.1  ROI  and  Improvement  Trend  Data: 


Table  4:  Investment  in  Quality  Tools 

Defect  tracking  system 


server 

software  licenses, 
25  no. 

Web  server 

server 

Software  testing  tools  for  quality 
(tools  like  Purify  /  bounds  checker 
/  Rational  Rose  /  Doors) 


systems 

software 

Total 


$35k 

$61.8k 


$30K 


$20K 
$50k 
$196.8  k 


4.26.2  Barriers  to  Achieving  High  Maturity 

•  High  growth  rate  needs  quick  ramp-up  of  people  towards  expertise  in  new  functional  and 
technological  domains. 

•  Pressure  for  continuous  ROI  for  quality  programs  through  COQ  reduction. 
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4.26.3  Unique  or  Distinguishing  Practices 

•  Six  Sigma — methodology  based  on  statistics  to  reduce  defects  in  software  and  meeting 
the  customer  Critical  to  Quality  needs  with  robust  design 

•  Digitization — to  webify  all  the  manual  processes  within  the  organization 

•  Complete  modality  integration  with  parent  organization 


4.26.4  People  and  Cuitural  Issues 

•  Turnover  is  high  at  25% 

•  There  is  an  extensive  induction  training  program  for  new  hires 

•  Team  building:  year-end  parties  (within  modality),  open  house  conducted  monthly 

•  Organization  focus  on  competence  development  and  hiring  top  talent 

•  Retention  of  talent — key  focus  area. 

•  Training  on  leadership  development 

4.26.5  Continuing  Improvement 

Focus  areas  for  2001: 

•  digitization 

•  software  reliability 

•  integration  of  Six  Sigma,  quality  system  (ISO,  FDA,  CE),  phase  review  discipline 


4.26.6  Summary 

•  Maturity  level  5  has  given  the  organization  a  definite  brand  equity  in  the  eyes  of  our  cus¬ 
tomers  and  also  been  responsible  for  attracting  top  talent. 

•  All  the  KPAs  well  integrated  with  ISO  9001,  IDA,  EN46001  into  a  single  software  life- 
cycle  process.  It  makes  life  easy  for  the  development  engineers  to  compliant  with  all  the 
above  requirements. 

•  Metrics  collection  is  seen  as  a  tedious  issue;  there  is  a  need  for  better  tools  for  automatic 
capturing  of  metrics.  We  need  to  re-look  into  the  right  metrics  which  have  more  business 
relevance  and  ensure  customer  satisfaction. 

•  Maturity  level  5  drives  a  data-based  culture  and  quantification  of  all  activities.  It  is  easy 
to  measure  the  capabilities  of  various  processes  and  the  outcome. 

•  Maturity  level  5  has  given  the  organization  a  well-defined  standard  process  to  be  fol¬ 
lowed  across  all  the  project  groups  to  identify  key  improvement  areas  and  deploy  Six 
Sigma  methodologies  in  achieving  them.  Level  5  helped  the  team  to  deliver  high  quality 
software  in  less  time. 
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4.27  Zensar  Technologies  Ltd,  Pune,  India 


Maturity  Level 
Date  of  Assessment 
Lead  Assessor(s) 

Web  Page 

Size  of  the  Organization 
Typical  Program  Size 

Primary  Application  Domain(s) 


5 

February  1999 
Richard  Knudson 
www.zensar.com 
1050  software  professionals 
10-12  people  per  project 
20-40  KLOC  per  project 
Systems  software 
Information  systems  software 
Outsourced  software 


The  organization  was  certified  to  ISO  9001  /  9000-3  in  the  year  1994. 

It  was  assessed  at  maturity  level  3  in  October  1996  and  maturity  level  5  in  February  1999. 
It  has  been  using  TQM  Frameworks  of  EFQM  since  1995  and  IQRS  since  2000. 


4.27.1  ROI  and  improvement  Trend  Data 

We  have  improvement  trends  in  product  quality,  where  processes  are  stable.  This  signifi¬ 
cantly  contributes  to  customer  satisfaction  and  therefore  customer  retention. 

We  meet  our  delivery  schedules  as  expected  by  the  customer. 

We  meet  our  profit  objectives,  where  we  do  projects  in  India. 

4.27.2  Barriers  to  Achieving  High  Maturity 

The  first  barrier  was  top  management  support  /  commitment.  This  support  needs  to  reflect  in 
the  amount  of  time  spent  on  these  activities  in  reviewing  the  progress  and  setting  process  im¬ 
provement  goals  that  support  business  goals. 

The  second  hairier  was  cultural,  related  to  discipline  at  work.  When  a  measurement  program 
is  implemented,  the  people  realize  that  if  they  provide  wrong  or  incomplete  data,  the  data 
analysis  shows  numbers  that  do  not  reflect  the  reality  and  therefore  it  is  useless.  Timely  col¬ 
lection  of  accurate  data  needs  discipline. 

4.27.3  Unique  or  Distinguishing  Practices 

Timely  collection  of  reasonably  accurate  data  from  the  process  which  is  good  enough  for  set¬ 
ting  performance  baselines. 
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Management  support  and  commitment  to  support  measurement  program. 

4.27.4  People  and  Cultural  Issues 

People  do  not  believe  in  or  like  to  support  measurement  programs.  They  believe  that  every 
project  is  too  different  from  other  projects  for  measurement  data  to  be  reused. 

However,  in  reality,  the  business  tries  to  get  projects  that  are  similar  to  past  projects  to  reduce 
risk,  and  over  a  period  it  is  possible  to  establish  that  measurements  add  value  for  project 
management  and  for  better  estimation. 

4.27.5  Continuing  Improvement 

We  have  been  using  TQM  models  since  1995.  Now  the  organization  is  more  focused  on 
overall  business  process  improvement. 

We  are  initiating  People  CMM  based  improvement  to  address  people  issues  better. 

We  are  also  studying  the  CMMI  and  have  plans  to  reassess  within  a  year’s  time. 

We  are  also  working  towards  certification  for  ISO  9000  (revision  2000). 

We  are  trying  to  become  more  customer  focused. 

4.27.6  Summary 

We  have  the  processes  /  tools  to  support  measurement  program.  This  supports  building  of  a 
project  database  and  library  which  is  a  real  asset  to  the  organization. 

Customer  expectations  have  gone  high  and  they  expect  improvements  in  productivity  and 
quality  over  a  period  of  time.  However,  it  is  difficult  to  achieve  this  if  the  customer  processes 
do  not  support  this.  This  is  especially  true  when  the  requirements  keep  changing  and  the  de¬ 
livery  dates  do  not  change. 

We  have  developed  in-house  training  programs  to  support  software  engineering  training 
needs  of  the  organization. 

Some  of  our  customers  are  engaging  us  for  consulting  assignments  to  improve  their  proc¬ 
esses. 
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5  Working  Groups 


Four  roles  were  recommended  for  each  working  group; 

1 .  a  facilitator,  to  make  sure  each  session  ran  smoothly 

2.  a  recorder,  to  take  detailed  notes  during  the  session 

3.  a  working  group  leader,  a  non-SEI  participant  to  make  the  presentation  at  the  general 
session  and  who  had  primary  responsibility  for  the  working  group  report 

4.  a  scribe,  to  capture  main  concepts  during  the  discussions  on  a  flip  chart  or  electronic 
projector 


Eleven  working  groups  were  proposed,  along  with  an  initial  set  of  questions  and  issues  to 
spark  discussion: 


Working  Group  1.1:  Measurement 

Working  Group  1.2:  Reliability  of  High  Maturity  Assessments 
Working  Group  1.3:  Six  Sigma  and  Software  CMM 
Working  Group  1.4:  Internet  Speed  and  Process  Improvement 
Working  Group  1.5:  CMMI 
Working  Group  2.1:  Statistical  Techniques 

Working  Group  2.2:  Business  Result  of  High  Maturity  Organizations 
Working  Group  2.3:  Change  Management  and  People  /  Cultural  Issues 
Working  Group  2.4:  Process  Agility 
Working  Group  2.5:  e-Commerce 
Working  Group  2.6;  Product  Lines 


After  the  review  of  the  proposed  working  groups  by  the  workshop  attendees,  the  working 
groups  on  business  results,  e-Commerce,  and  product  lines  were  cancelled.  The  working 
group  on  CMMI  was  duplicated  in  both  of  the  workshop  sessions  because  of  high  interest. 
The  working  groups  on  Internet  speed  and  process  agility  were  combined. 
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5.1  Working  Group  1.1  -  Measurement 

The  participants  in  this  working  group  included 

•  Phil  Sperling  (working  group  leader  and  presenter),  Telos-OK 

•  Dennis  Goldenson  (facilitator),  SEI 

•  Wendy  Irion-Talbot  (scribe  and  report  lead),  CSC 

•  Dave  Zubrow  (recorder),  SEI 

•  Joan  Romine,  Boeing 

•  Jitendra  Shreemali,  Philips  Software  Centre  Limited 

•  Mel  Wahlberg,  CSC 

•  Jim  Vanfleet,  USAF  /  Hill  /  TIS-3 

•  Somashekhar  Ramadevanahalli,  CG-Smith  Software  Ltd. 

•  Subrata  Guha,  Satyam  Computer  Services  Ltd. 

5.1.1  Questions,  Observations  and  Hypotheses 

The  initial  questions  put  forth  by  the  workshop  committee  and  group  members  included: 

•  Do  measures  change  substantively  as  organizations  move  from  Level  3  to  Level  4?  How 
does  organizational  size  affect  this,  if  at  all? 

•  Are  the  changes  in  measurement  a  difference  in  granularity,  e.g.,  collecting  process  data 
at  the  activity  level  as  opposed  to  phase  level,  or  a  difference  in  kind,  e.g.,  moving  from 
defect  density  data  to  mean-time-to-failure  data  for  measuring  quality? 

•  Using  measures  as  part  of  the  causal  analysis  process  can  be  challenging.  To  what  level 
of  granularity  must  causal  analysis  be  carried?  At  what  point  does  an  organization  cease 
causal  analysis  of  an  issue? 

•  What  processes  or  subprocesses  are  typically  being  “quantitatively  managed”  at  Levels  4 
and  5  and  what  are  the  associated  measures  that  are  being  tracked? 

•  What  are  the  [measurement-based]  decisions  made  by  high  maturity  organizations?  What 
needs  to  be  considered?  What  analytical  techniques  are  appropriate  beyond  SPC? 

•  Once  you’ve  reached  Level  5,  what  are  the  next  steps  and  challenges  [in  a  measurement 
context]? 

•  How  do  you  identify  the  right/appropriate  data  for  “organizational  metrics?” 

•  How  have  other  organizations  achieved  Level  4? 

•  Is  the  Software  CMM  (version  1.1)  addressing  the  right  measurement  issues? 

During  introductions,  working  group  members  shared  the  following  observations  and  hy¬ 
potheses  as  we  got  started. 
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•  A  disconnect  often  exists  between  measures  required  for  management  and  those  neces¬ 
sary  to  meet  project  needs  on  a  variety  of  levels.  For  example: 

-  Senior  management  cares  about  resource  loading  factors  and  planned  vs.  actual  ex¬ 
penditures. 

-  An  SEPG  cares  about  process  quality  measures,  estimation,  and  effort  and  schedule 
variances. 

-  A  project  leader  cares  about  requirements  stability,  and  effort. 

In  addition,  there  are  also  concerns  about  the  costs  associated  with  the  collection  and  re¬ 
porting  of  measures  (particularly  redundant  measures),  especially  up  the  management 
chain  where  the  utility  of  some  data  is  not  clear  and  frequent  changes  in  direction  can  be 
costly  and  frustrating.  Aggregating  data  to  some  level  of  abstraction  may  be  inappropri¬ 
ate.  For  example,  rolling  up  to  the  enterprise  level  may  not  make  sense  for  all  measures, 
even  when  requested  by  senior  management. 

How  do  you  reconcile  and  balance  these  perspectives  and  issues?  Consider  that  the  pro¬ 
ject  is  the  sole  source  of  data  for  measures! 

•  A  common  set  of  metrics  can  be  defined  at  the  organizational  level,  perhaps  by  the  SEPG, 
but  also  may  be  met  with  resistance  and  may  not  prove  to  be  useful.  Don’t  try  to  stan¬ 
dardize  measures  across  too  large  an  organization.  In  heterogeneous  organizations  the 
common  metrics  approach  across  projects  may  not  be  at  all  successful.  A  better  approach 
might  be  to  identify  conunon  categories  with  specific  common  measures.  Some  organiza¬ 
tions,  perhaps  with  more  homogeneous  projects,  had  some  success  standardizing  on 
measures  that  enabled  resources  to  move  between  projects,  increasing  efficiency  and  sav¬ 
ing  dollars. 

•  Changing  definitions  of  measures  degrades  the  utility  of  historical  databases,  baselines, 
and  parametric  models,  and  makes  automation  of  data  collection  difficult.  The  impact  of 
changes  in  the  definitions  of  measures  can  be  sweeping.  For  example,  changes  to  the 
definition  of  a  line  of  code  affect  measures  of  total  system  size  and  all  derived  indicators 
using  this  measure,  and  make  comparisons  with  historical  data  difficult  or  invalid.  Within 
the  DoD,  new  definitions  have  just  been  promulgated  from  the  Navy  (PEO-TSC).  Some 
projects  have  data  going  back  30  years  with  many  releases,  high  reuse,  and  historical  de¬ 
fect  densities.  While  the  PEOs  are  trying  to  standardize  across  their  programs,  which  is  a 
useful  approach  at  a  high  level,  individual  projects  are  now  challenged  to  recalibrate  to 
the  new  definitions,  and  lose  some  utility  of  historical  data  and  models. 

Measurement  definitions  may  influence  how  programmers  choose  to  format  their  code, 
e.g.,  use  of  carriage  returns,  comments,  indention.  Changing  definitions  may  or  may  not 
cause  corresponding  changes  in  code  formats,  and  make  calibration  or  factoring  of  his¬ 
torical  data  inconsistent.  This  calibration  can  further  be  impacted  if  measurement  defini¬ 
tion  is  accompanied  by  a  corresponding  tool  change. 

These  changes  also  impact  comparison  with  others,  tracking  process  improvement,  and 
reporting  to  senior  management.  This  can  be  especially  troublesome  when  organizations 
come  together  to  partner,  as  for  a  proposal,  and  measurement  definitions  are  disparate 
and  can’t  be  easily  normalized. 

•  Core  data  collected  doesn’t  appear  to  change  that  much  at  maturity  level  4  and  5.  Rather, 
the  derived  metrics,  sophistication  of  analysis,  presentation  and  visualization,  and  rele¬ 
vance  to  decision  making  becomes  more  useful  to  the  organization  for  control  and  im¬ 
provement  purposes.  However,  “who”  had  been  doing  the  measurement  (possibly  devel- 
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oping  additional  derived  measures)  and  analysis  possibly  shifted  or  expanded  across  roles 
as  maturity  advanced. 

•  Across  the  organizations  represented  in  the  working  group  there  was  a  great  deal  of 
commonality  in  the  data  collected.  Some  groups  had  normalized  on  six  core  measures 
and  others  as  many  as  70  that  reduced  to  a  balanced  score  card  for  presentation,  but  most 
fell  within  common  boundaries,  e.g.,  cost,  schedule,  effort,  size,  changes,  defects,  etc. 

•  Managing  the  culture  change  as  an  organization  seeks  to  move  to  Level  4  seemed  to  be  a 
widespread  issue.  There  are  challenges  in  perception,  especially  where  practitioners  had 
just  enough  Software  CMM  literacy  to  “be  dangerous”  (e.g.,  “Measurement  is  only  for 
high  maturity  organizations.  I  just  want  to  be  Level  3,  so  I  don’t  have  to...”)  and  actually 
gives  lower  maturity  projects  a  reason  not  to  focus  on  measurement.  There  are  challenges 
in  understanding  model  requirements,  for  example  probing  may  reveal  the  project  is  in¬ 
deed  making  decisions  based  on  defined  thresholds,  but  did  not  understand  the  model’s 
application  in  their  context. 

•  A  variety  of  experiences  and  “dos”  and  “don’ts”  were  surfaced  that  the  group  decided  to 
discuss  in  more  detail.  These  are  captured  below  in  the  “Pitfalls  and  False  Starts”  section. 


There  were  many  questions  and  issues  raised  by  the  group,  but  the  timebox  allotted  to  each 
working  group  session  did  not  allow  exploration  and  resolution  of  all  of  them.  Two  issues 
that  the  working  group  decided  to  discuss  in  more  detail  were 

1.  Change  management  as  applied  to  measurement 

2.  Pitfalls  and  false  starts 

5.1.2  Change  Management  as  Applied  to  Measurement 

Change  management  was  considered  to  be  a  key  issue  in  moving  beyond  Level  3.  Most 
working  group  participants  cited  examples  where  the  effort  to  promulgate  the  culture  change 
needed  to  move  from  Level  3  to  Level  4  was  at  least  as  challenging  as  the  technical  issues 
associated  with  quantitative  management.  The  following  key  points  were  raised  in  associa¬ 
tion  with  this  issue: 

1.  Nurturing  the  workforce  about  higher  maturity  concepts  is  essential.  Working  group 
participants  pointed  out  that  the  core  data  collected  at  Levels  4  and  5  really  did  not 
change  much.  Rather,  how  it  was  applied  and  by  whom  and  what  additional  measures 
were  derived  from  the  core  set  changed.  This  change  requires  that  the  practitioners  un¬ 
derstand  the  premise  and  intended  application  and  impact  of  measures,  especially  from  a 
quantitative  management  perspective.  This  nurturing  can  take  many  forms,  and  includes 
training,  coaching  and  mentoring. 

2.  Broadening  the  involvement  of  the  organization,  vertical  and  horizontal,  with  re¬ 
spect  to  measurement  and  analysis  was  also  identified  as  essential.  High  maturity  or¬ 
ganizations  had  very  broad  involvement  of  staff  and  management  across  their  projects. 
One  group  characterized  the  change  in  senior  management  commitment  as  “enabling” 
budget  and  resources  at  Levels  2  and  3,  whereas  at  Levels  4  and  5  it  becomes  “personal” 
leadership,  knowledge,  direct  participation  and  is  the  basis  for  decision-making.  This 
particular  change  was  cited  as  key  to  their  success  in  achieving  Levels  4  and  5.  Partici- 
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pation  across  the  various  functional  groups  in  the  organization  was  also  prevalent.  No 
longer  was  the  SEPG  trying  to  drive  everything,  but  functional  groups  seemed  to  be 
more  involved  in  measurement  collection  and  analysis,  and  also  use  that  as  the  basis  for 
decision  making. 

3.  Organizations  cannot  stay  at  maturity  level  3 — they  will  either  advance  (use  the 
data)  or  regress  (quit  measuring).  The  working  group’s  feeling  was  that  Level  3  was 
inherently  unstable.  Advancing  organizations  apply  the  data  or  derive  additional  meas¬ 
ures  to  control  and  improve  processes  across  functional  areas  or  project  phases.  Groups 
that  don’t  take  this  next  step  tend  to  stop  doing  other  than  basic  cost  /  schedule  measur¬ 
ing  and  hence  regress.  “Just  enough”  Software  CMM  literacy  to  be  dangerous — 
measurement  is  not  just  for  high  maturity  organizations  (senior  managers,  project 
managers,  customers).  This  point  really  unites  the  change  management  issues  funda¬ 
mental  to  advancing  the  maturity  of  an  organization.  Practitioners  need  to  understand 
that  measurement  is  essential  to  knowing  where  you  are  with  respect  to  your  program 
and  process  goals,  and  this  understanding  begins  with  the  first  KPA  at  the  lower  maturity 
levels.  They  also  need  to  understand  advanced  maturity  concepts,  approaches  and  bene¬ 
fits,  and  then  they  need  to  be  involved.  Comments  indicating  that  measurement  and 
analysis,  which  really  begins  with  basic  performance  measures  collected  at  Level  2,  is 
only  for  “high  maturity  organizations’’  are  a  clear  signal  that  education  groundwork  is 
needed. 


5.1.3  Pitfalls  and  False  Starts 

Working  group  members  were  curious  as  to  whether  their  false  starts  were  common  and  how 
the  other  groups  represented  handled  those  challenges,  and  further  felt  that  sharing  their  les¬ 
sons  learned  could  be  beneficial  to  other  organizations  seeking  to  advance  their  maturity.  The 
following  four  points  were  raised  in  association  with  this  issue: 

1 .  Choosing  the  wrong  measures  is  one  of  the  biggest  potential  pitfalls.  Organizations 
need  to  understand  the  advanced  maturity  concepts  and  underlying  measurement  ration¬ 
ale  first,  then  determine  what  measures  are  appropriate  for  them.  Focus  on  the  “why”  do 
we  want  to  collect  them,  then  define  “what”  needs  to  be  collected  followed  by  “how”  to 
perform  the  analysis  and  decision  making.  Be  wary  of  silver  bullets  and  packaged  solu¬ 
tions — what  worked  for  another  organization  may  not  be  directly  applicable  to  you  in 
other  than  a  general  way. 

2.  Disconnects  between  project  goals  and  process  improvement  goals  can  cause  per¬ 
petual  confliction  of  priorities  and  prohibit  understanding  and  progress.  Not  in¬ 
volving  the  stakeholders  (managers,  customers,  and  workforce)  in  the  design  and  pur¬ 
pose  of  the  measures  could  be  fatal  to  the  success  of  your  [quantitative]  measurement 
program.  Several  examples  were  discussed  in  which  the  SEPG  and  other  process  re¬ 
sources  were  given  goals  like  “get  to  Level  4”  and  managers  were  given  goals  like  “con¬ 
tain  cost  and  schedule.”  This  separation  of  goals  led  to  frustrating  conflicts  in  resource 
use  and  task  priorities.  Success  was  not  achieved  until  each  of  the  goals  was  understood, 
the  relationship  between  them  was  understood,  and  they  each  became  goals  for  every¬ 
one.  It’s  like  a  visit  to  the  chiropractor,  once  the  elements  were  in  alignment,  the  organi¬ 
zation  as  a  whole  was  able  to  function  more  effectively  and  with  less  pain.  Beware  the 
SEPG  trying  to  do  it  all  themselves  and  not  getting  involved  in  projects.  Not  only  is 
there  the  danger  of  the  “Ivory  Tower  Syndrome,”  but  very  often  the  SEPG  is  spread  very 
thin,  and  they  are  not  necessarily  directly  responsible  for  project  (product)  work.  In 
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Level  2  and  3  organizations  you  very  often  see  the  SEPG  assigned  as  the  clear  focal 
point  for  all  process  improvement,  particularly  from  the  OPD  /  OPF  perspectives.  This 
expectation  tends  to  remain  as  the  organization  looks  to  Levels  4  and  5.  But,  with  ad¬ 
vancing  maturity,  it  is  natural  and  appropriate  to  involve  more  and  more  resources  across 
the  projects  and  organization  directly  in  process  and  quantitative  management.  The 
SEPG  remains  the  focal  point  for  some  process-based  activities,  but,  more  and  more,  the 
advanced  maturity  institutionalizes  ownership  and  participation  throughout  their  organ¬ 
izational  structure  and  the  SEPG  moves  into  a  facilitator  role.  This  is  appropriate  as 
analysis  and  decision  making  at  all  levels  becomes  based  on  the  quantitative  techniques. 
Bottom  line:  the  organization  moves  to  take  ownership  of  the  processes  and  process 
improvement  initiatives  as  part  of  its  activities,  rather  than  as  separate  initiatives. 

Some  projects  have  successfully  used  “process  consultants”  as  a  way  to  deploy  and  in¬ 
volve  process  resources  across  the  project(s),  providing  support  and  guidance,  and  ena¬ 
bling  project  resources  to  execute  process  initiatives. 

4.  Citing  irrelevant  examples  during  deployment  and  implementation  of  QPM  and 
SQM  can  throw  more  obstacles  in  your  way.  The  most  effective  analogies  and  exam¬ 
ples  are  those  that  relate  directly  to  the  group  with  which  you  are  working.  Picking  arbi¬ 
trary  examples  may  not  enable  understanding,  and  actually  may  further  distance  (or  dis¬ 
courage)  the  target  audience  from  the  goal  at  hand. 


5.1.4  Recommendations  for  Organizations  Seeking  High 
Maturity 

As  a  result  of  the  working  group’s  discussions,  the  following  recommendations  for  organiza¬ 
tions  seeking  to  achieve  high  maturity  were  identified  as  “critical  factors  for  success”: 

•  Involve  those  impacted  by  the  measurement  program  is — do  it  with  them,  not  to  them. 
The  importance  of  understanding  and  buy-in  at  all  levels  cannot  be  understated.  Work 
with  project  resources  to  understand  what  they  are  already  doing.  Guide  them  through  us¬ 
ing  these  and  additional  primary  or  derived  measures  that  can  be  used  to  more  effectively 
address  their  problems,  and  eventually  to  predict  and  prevent  problems.  In  addition,  the 
approach  to  using  the  measurements  should  become  proactive  rather  than  reactive  (pre¬ 
dictive  rather  than  correcting). 

•  Start  with  small,  focused  efforts  to  generate  some  early  successes.  Mandates,  especially 
those  generated  external  to  the  project,  are  rarely  successful. 

•  Integrate  project  measures  and  business  objectives.  Failure  to  do  so  will  probably  pre¬ 
vent  success.  Use  the  Goal-Question-Measure  method  as  a  starting  point,  beginning  at  the 
highest  level  with  your  business  goals.  Ensure  you  show  and  reinforce  your  chosen  ap¬ 
proach  and  relevance  to  your  project  and  organization.  Comments  such  as  “Thanks  for 
the  meeting,  now  I  have  to  go  do  real  work”  are  clear  signals  that  the  goals  are  not  yet 
aligned  and  people  still  feel  there  is  a  gap  between  process  and  project  priorities. 

•  Revisit  the  basics  and  review  the  purpose  and  need  for  each  measure.  This  can  and 
should  be  done  early  (even  at  Level  2  for  those  measures)  and  reinforced  as  the  organiza¬ 
tion  advances  maturity.  See  the  CMMI  Measurement  and  Analysis  PA. 

•  Automate  collection  and  analysis  as  much  as  practicably  possible.  Invest  in  a  statistical 
package  if  you  can.  This  will  not  only  streamline  the  measurement  process  and  ensure 
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consistency  of  data  and  collection,  but  will  enable  people  to  focus  on  the  analysis  and  de¬ 
cision  processes  rather  than  expend  effort  on  the  mundane  collection  tasks. 

•  Address  change  management  as  applied  to  measurement  in  the  context  of  high  maturity. 
As  noted  in  the  discussion  above,  change  management  and  consideration  of  the  cultural 
issues  in  moving  beyond  Level  3  are  even  more  important  than  the  high  maturity  organi¬ 
zations  had  realized  at  the  outset.  Determining  the  technical  requirements  for  a  quantita¬ 
tive  process  management  system  is  a  tremendous  task,  but  if  corresponding  people  issues 
are  not  handled,  you  run  the  risk  of  building  an  ineffective  Level  4/5  infrastructure.  In 
addition,  as  you  tackle  and  achieve  success  in  small  pockets  of  the  organization,  commu¬ 
nicate  the  successes  and  benefits  of  moving  to  Levels  4  and  5.  Leverage  the  pockets  of 
excellence  and  learn  how  to  adjust  and  adapt  these  to  other  areas  of  the  organization. 

5.1 .5  Recommendations  for  the  SEI 

As  a  result  of  the  working  group’s  discussions,  the  following  recommendations  were  formu¬ 
lated.  The  SEI  needs  to: 

•  Encourage  organizations’  understanding  of  measurement  at  lower  maturity  levels. 
Influence  the  perception  that  measurement  only  applies  to  the  higher  maturity  areas.  Em¬ 
phasize  that  it  applies  to  everyone  at  some  level.  Organizations  should  establish  the  foun¬ 
dations  of  their  measurement  program  early  on  to  generate  understanding  and  begin  to 
build  historical  data  sets.  This  will  assist  organizations  in  avoiding  false  and  ineffective 
starts.  Avoid  false  starts;  start  early,  start  small,  and  be  real. 

•  Address  change  management  as  applied  to  measurement  in  the  context  of  high  ma¬ 
turity.  Allow  for  the  time  and  effort  necessary  to  effect  the  culture  changes  that  must  ac¬ 
company  a  move  up  from  Level  3. 

•  Provide  guidelines  and  examples  for  quantitative  analysis  techniques  and  methods 
that  are  acceptable  at  Level  4  (other  than  SPC).  Group  members  agreed  that  not  all 
processes  could  or  should  be  quantitatively  controlled  using  SPC — it  just  doesn’t  make 
sense.  The  community  could  benefit  from  additional  guidance. 

A  useful  mechanism  to  address  these  recommendations  might  be  to  augment  training  for  high 
maturity  organizations  to  provide  additional  focus  and  guidance  in  these  areas. 

5.2  Working  Group1.2:  Reliability  of  High  Maturity 
Assessments 

The  participants  in  this  working  group  included  Roger  Bate  (SEI),  Donna  Dunaway  (SEI), 
Will  Hayes  (SEI),  Bill  Hefley  (Q-Labs),  Gargi  Keeni  (Tata  Consultancy  Services),  Linda  Le¬ 
vine  (SEI),  Judah  Mogilensky  (Process  Enhancement  Partners),  Joseph  Morin  (Integrated 
System  Diagnostics),  Muthuramalingam  Rajamanickam  (HCL  Technologies),  David  White 
(SEI),  and  John  Yu  (Motorola-China). 

The  working  group  leader  was  Judah  Mogilensky  (Process  Enhancement  Partners).  The  fa¬ 
cilitator  was  Will  Hayes  (SEI).  The  scribe  was  Bill  Hefley  (Q-Labs).  The  recorders  were 
Donna  Dunaway  (SEI)  and  Joseph  Morin  (Integrated  System  Diagnostics). 
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5.2.1  Points  of  Departure 

The  working  group  had  the  benefit  of  the  report  from  the  working  group  on  the  same  topic 

from  the  previous  High  Maturity  Workshop,  conducted  in  November  1999.  (See  [Paulk  00a].) 

Among  the  suggestions  put  forward  in  that  report  (page  46).  and  observations  by  the  Working 

Group  about  what  has  happened  since  the  last  workshop,  were  the  following: 

•  Re-establish  the  CMM  Advisory  Board.  [Not  done.] 

•  Establish  mandatory  supplemental  training  for  any  Lead  Assessor  to  lead  Level  4  or  5 
assessments.  [Not  mandatory  at  present,  but  the  SEI  does  now  offer  High  Maturity  with 
Statistics  course  and  the  SPC  for  Software  course.] 

•  Gather  data  regarding  the  alleged  problem  of  inconsistent  or  inappropriate  Level  4  and  5 
assessment  results.  [Assessment  findings  briefings  and  reports  are  collected,  but  there  has 
been  no  program  of  quality  analysis  of  the  reported  findings  for  Level  4  and  5  KPAs.  It 
was  noted  that  there  is  currently  no  such  analysis  of  reported  findings  at  lower  levels,  ei¬ 
ther.] 

•  Periodically  conduct  High  Maturity  Practices  Workshops.  [On  the  one  hand,  the  second 
such  workshop  is  now  being  held  within  two  years.  On  the  other  hand,  there  is  very  low 
participation  from  Lead  Assessors  in  this  workshop.] 

•  Elicit  papers  on  high  maturity  topics  from  the  community  at  large.  [It  was  noted  that  the 
last  two  SEPG  conferences  have  had  a  significantly  larger  number  of  high  maturity  pres¬ 
entations  than  in  earlier  years,  and,  furthermore,  these  sessions  have  been  quite  well  at¬ 
tended.] 


The  questions  that  inspired  the  creation  of  this  working  group  included: 

•  Are  a  significant  percentage,  perhaps  even  a  majority,  of  the  organizations  assessed  at 
Levels  4  or  5  really  just  good  Level  3  organizations? 

•  What  are  the  typical  misinterpretations  that  lead  to  inflated  level  ratings? 

•  What  qualifications  should  Lead  Assessors  or  Evaluators  have  to  perform  high  maturity 
appraisals? 

•  How  can  we  equitably  evaluate  Lead  Assessors  or  Evaluators  to  demonstrate  their  ability 
to  judge  higher-level  implementations?  Curriculum  of  accepted  training?  An  examina¬ 
tion?  Specialized  certification  through  a  body  such  as  ASQ? 

•  What  qualifications  should  an  assessment  team  have  to  perform  a  high  maturity  assess¬ 
ment? 

•  Could  the  community  support  increased  team  training  for  higher  maturity  levels? 

•  What  are  the  hard  decisions  as  an  assessor? 

•  What  are  the  difficult  things  to  get  teams  to  understand? 

•  What  are  the  most  difficult  findings  to  explain  to  an  organization? 

•  What  additional  training,  auditing,  etc.,  should  the  SEI  perform  to  retain  the  credibility  of 
high  maturity  assessments? 
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Do  the  CMMI  models  contain  information  that  will  help  assure  reliability,  or  might  it 
lead  to  reduced  reliability? 

Are  there  changes  we  can  suggest  to  CMMI  that  will  enhance  reliability? 


5.2.2  Topics  Selected  by  the  Working  Group  to  Focus 
Discussion 

After  a  discussion  of  the  points  of  departure  and  the  concerns  of  the  participants,  consensus 
emerged  that  the  fundamental  concern  of  the  working  group  was  the  perception  within  the 
process  improvement  community  (which  may  or  may  not  be  based  on  reality)  that  unaccept¬ 
able  variation  exists  in  the  determination  of  Level  4  and  5  appraisal  ratings.  This  perception 
is  not  inherently  restricted  to  any  particular  model  or  appraisal  method. 


This  concern  was  based  upon  four  key  underlying  issues,  namely 

1.  The  available  descriptive  models  (to  varying  degrees)  do  not  specify  what  is  required  to 
achieve  Levels  4  and  5  in  an  exhaustive  and  unambiguous  manner. 

2.  The  community  has  no  well-accepted  and  widely  available  consensus  as  to  what  is  actu¬ 
ally  required  for  achievement  of  these  ratings.  Further,  there  is  no  recognized  forum  for 
achieving  the  desired  consensus. 

3.  Currently,  Lead  Appraiser  programs  do  not  include  specific  qualification  requirements 
for  Levels  4  and  5,  specific  model  and  method  training  requirements  for  these  levels,  or 
corresponding  training  materials  and  implementation  guides.  Training  materials  and  im¬ 
plementation  guides  are  also  lacking  for  those  organizations  attempting  to  achieve  these 
levels. 

4.  There  is  no  comprehensive  mechanism  functioning  to  validate  the  accuracy,  quality, 
consistency,  and  reliability  of  Level  4  and  5  appraisal  results.  Similarly,  there  is  no 
mechanism  functioning  to  ensure  that  appraisal  teams  perform  to  any  minimum  stan¬ 
dards. 


The  working  group  settled  on  the  following  main  topics  to  discuss  in  more  detail: 

•  Qualification  and  Preparation  of  Lead  Appraisers  and  Teams 

-  qualifications 

-  training  (both  initial  and  sustaining) 

•  Development  and  Dissemination  of  Public  Guidance  for  Implementation  and  Inter- 
pretation  of  Level  4  and  Level  5  KPAs 

-  assessment  guidance 

-  implementation  guidance 

-  interpretations  or  common  misinterpretations 

•  Achievement  of  Greater  Reliability  for  High  Maturity  Appraisals 

-  analyzing  assessment  data 

-  quality  assurance 
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-  reassessments  (especially  after  achieving  a  Level  5  rating) 

“  approach;  deployment;  results 

The  working  group  recognized  that,  while  these  topics  are  to  be  discussed  separately,  they  are 
very  much  tied  together  in  terms  of  addressing  the  issues  identified  above.  Thus,  the  reader 
will  note  significant  overlap  among  the  three  main  topics. 

5.2.3  Qualification  and  Preparation  of  Lead  Appraisers  and 
Teams 

The  group  addressed  the  first  topic  using  a  structured  format:  Why  is  this  an  issue?  What  is 
the  current  status?  What  are  the  implications  and  significance  of  the  current  status?  What  are 
some  opportunities  to  improve  (i.e.,  recommendations)  that  the  group  would  like  to  make? 


5.2.3.1  Why  is  This  an  issue? 

Reasons  identified  by  the  group  for  why  preparation  of  leads  and  teams  was  an  issue  for  high 

maturity  appraisals  included  the  following: 

•  The  community  has  a  need  for  consistency  /  reliability  of  assessment  results. 

•  Appraisals  are  facing  increased  complexity;  multiple  models,  multiple  disciplines. 

•  There  seems  to  be  no  broad  community  consensus  on  the  specific  requirements  that  an 
organization  must  satisfy  to  deserve  a  rating  of  maturity  level  4. 

•  There  currently  exists  a  broad  range  of  interpretation  for  the  existing  Software  CMM 
vl.l  high  maturity  KPAs,  encouraged  somewhat  by  the  vagueness  of  the  text;  currently 
required  Software  CMM  training  for  both  leads  and  teams  is  the  SETs  Introduction  to  the 
CMM  course,  which  is  generally  viewed  as  providing  inadequate  coverage  of  Levels  4 
and  5. 

•  Organizations  wishing  to  conduct  serious  benchmarking,  especially  for  demonstrating 
continuous  improvement,  may  find  these  efforts  more  difficult. 

•  There  are  Lead  Assessors,  Lead  Evaluators,  and  Assessment  Team  Members  who  are  in¬ 
experienced  in  maturity  level  4  and  5  topics,  and  who  may  confront  these  topics  without 
adequate  preparation. 

•  The  community  desires  assessment  results  to  be  more  useful  and  less  subject  to  dispute. 

•  Sharing  experiences  will  improve  the  LA  /  LE  community. 

•  Assessors  are  being  questioned  about  how  they  conduct  appraisals,  and  about  the  results 
of  those  appraisals,  because  appraisal  method  requirements  and  assessor  qualifications 
are  not  well  understood. 

•  Lead  Assessors  and  Lead  Evaluators  may  have  to  learn  multiple  disciplines  to  work  with 
the  CMMI  models. 
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5.2.3.2  Current  Status 

Aspects  of  the  current  status  highlighted  by  working  group  members  included: 

•  Assessments  are  longer  and  more  painful  if  supplementary  on-the-spot  Assessment  Team 
Member  training  on  Level  4  and  5  KPAs  is  required  during  the  on-site  period. 

•  Expectations  are  not  well  established  for  the  qualifications  of  Lead  Assessors  and  Lead 
Evaluators  to  address  Level  4  and  5  KPAs. 


5.2.3.3  Implications/Significance 

Key  implications  and  aspects  of  the  significance  of  the  current  status  in  this  area,  as  identified 

by  the  working  group  members,  were  as  follows: 

•  Training  is  also  needed  for  organization,  not  just  for  the  Lead  Assessors,  Lead  Evalua¬ 
tors,  and  team  members;  organizations  lack  understanding  of  how  to  implement  maturity 
level  4  and  5  process  areas. 

•  There  is  a  need  to  beef  up  expectations  (LA,  ATMs)  regarding  what  it  means  to  be  ade¬ 
quately  trained  and  qualified  to  conduct  high  maturity  appraisals. 

•  The  current  status  results  in  potentially  inconsistent  assessments  and  promoting  “claim¬ 
ing”  levels,  rather  than  focusing  on  real  benefits.  In  other  words,  it  enables  “level¬ 
grabbing,”  with  negative  impacts  on  teams  and  on  assessment  conduct. 

•  The  current  status  may  cast  doubt  upon  any  maturity  level  4  or  5  assessment,  including 
the  absolutely  valid  ones,  resulting  in  poor  perception  of  the  model,  assessments,  etc. 

•  Organizations  invest  a  lot  in,  and  expect  a  lot  of,  assessments,  so  they  should  not  have  to 
be  concerned  about  questions  regarding  the  validity  of  the  results. 

•  Results  of  assessments  are  at  risk  if  the  quality  of  the  Lead  Assessor  or  Lead  Evaluator  is 
not  known  to  the  organization  being  appraised,  or  to  the  users  of  appraisal  results. 


5.2.3.4  Opportunities  to  Improve 

As  already  noted,  the  report  from  the  November  1999  High  Maturity  Workshop  listed  several 
recommendations,  most  of  which  have  not  yet  been  implemented.  In  addition  to  these,  the 
working  group  members  noted  the  following  suggestions: 

•  Encourage  Lead  Assessors  /  Lead  Evaluators  to  take  additional  training  in  high  maturity 
topics  (e.g.,  current  SEI  courses  in  SPCfor  Software  and  in  High  Maturity  with  Statistics, 
as  well  as  the  Intermediate  Concepts  of  the  CMMI  course,  now  required  for  CMMI  Lead 
Assessors) 

•  Suggest  appraisers  use  a  standard  experience  /  qualification  matrix  as  a  means  for  Lead 
Assessors  and  Evaluators  to  describe  their  level  and  scope  of  experience  (maturity  levels, 
disciplines,  etc.)  and  for  appraisal  customers  to  better  match  LA  /  LE  qualifications  to  the 
parameters  of  their  appraisal.  (The  working  group  noted  that  all  attempts  in  the  past  to  es¬ 
tablish  different  “classes”  of  Lead  Assessors  and  Evaluators  had  failed,  so  we  will  not 
suggest  that  approach  again.) 
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•  Provide  additional  model  and  method  implementation  guidance  for  appraisers  (which 
will  also  serve  as  useful  input  to  organizations  seeking  appraisals)  in  such  areas  as: 

-  What  evidence  is  indicative  of  implementation  and  institutionalization  of  higher  ma¬ 
turity  practices? 

-  How  do  organizations  produce  appropriate  observations  and  findings  with  respect  to 
high  maturity  practices  and  goals? 

-  What  circumstances  trigger  the  need  for  a  new  baseline  appraisal? 

•  Utilize  newly  available  mechanisms  for  disseminating  this  guidance  to  the  lead  appraiser 
community  (for  example,  the  new  SEI  Lead  Assessor  Web  site,  the  International  Associa¬ 
tion  of  Professional  Lead  Assessors  [lAPLA]  Web  site). 


5.2.4  Development  and  Dissemination  of  Public  Guidance 

The  working  group  chose  not  to  have  the  same  type  of  structured  discussion  for  this  second 
topic.  Instead,  the  group  chose  to  address  the  topic  in  a  more  free-form  manner. 

The  primary  issue  underlying  this  second  topic  is  that  the  process  improvement  community 
lacks  a  recognized  and  accepted  mechanism  for  establishing  consensus  on  model  interpreta¬ 
tion  and  implementation  issues.  (This  issue  is  the  basis  for  the  suggestion  in  the  November 
1999  workshop  report  to  re-establish  the  CMM  Advisory  Board.  The  CAB  served  as  just  such 
a  mechanism  for  establishing  interpretation  consensus  for  the  Software  CMM  vl.l.) 

Nevertheless,  it  would  be  valuable  for  the  community  to  have  consensus  guidance  both  for 
implementation  and  for  appraisals  in  such  areas  as: 

•  What  does  quantitative  management  really  mean  (what  aspects  must  be  managed  quanti¬ 
tatively,  how  to  judge  the  suitability  of  the  quantitative  methods  being  used,  etc.)? 

•  What  constitutes  effective  defect  prevention  activities  (how  should  defect  classes  be  se¬ 
lected  for  root  cause  analysis,  how  often  should  root  cause  analysis  be  conducted,  should 
defect  prevention  process  changes  be  made  just  on  a  project  or  phase  basis  or  must  they 
be  organization-wide,  how  much  follow-up  evaluation  should  be  performed  for  defect 
prevention  process  changes,  etc.)? 

•  What  constitutes  effective  continuous  improvement  (how  should  pilot  trials  of  new  tech¬ 
nology  and  new  processes  be  conducted,  how  much  data  should  be  collected,  how  are 
data  from  pilot  trials  to  be  fed  back  into  the  OSSP,  how  much  overall  improvement  and 
process  change  activity  must  be  taking  place,  etc.)?  [Please  note  that  this  topic  is  also  ad¬ 
dressed  in  the  Working  Group  Report  for  WG  2.3,  Change  Management  and  People  / 
Cultural  Issues  in  High  Maturity  Organizations.] 

•  How  do  quantitative  management  and  continuous  improvement  clearly  and  visibly  con¬ 
nect  to  business  goals  and  objectives? 


The  working  group  developed  a  general  outline  of  the  format  that  this  guidance  should  take. 
The  format  would  consist  of  four  key  sections: 

1 .  A  description  of  the  topic,  abstracted  from  multiple  relevant  models 
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2.  Interpretive  guidance  for  organizational  implementation  and  for  appraisals  (e.g.,  rules  of 
thumb,  examples  of  successful  implementations,  non-attribution  examples  of  specific 
judgments  made  by  assessment  teams,  etc.) 

3.  Commonly  encountered  problems,  misinterpretations,  and  misconceptions  (including 
examples  of  misunderstandings  from  High  Maturity  Practices  surveys  and  instances  of 
apparent  errors  in  assessment  findings) 

4.  References  for  additional  guidance,  including  mappings  to  specific  model  content  and  an 
annotated  bibliography  of  supplemental  resources  (texts,  tools,  etc.) 


Although  the  working  group  made  no  specific  recommendation  for  a  body  to  take  responsi¬ 
bility  to  develop  this  interpretive  guidance,  conduct  broad  public  reviews  of  it,  and  then  make 
it  generally  available  to  the  process  improvement  community,  it  was  noted  that  a  number  of 
existing  organizations  could  potentially  contribute  to  these  activities,  including  the  SEI, 
lAPLA,  ASQC  Software  section,  IEEE  Computer  Society,  ACM,  NDIA,  EIA,  INCOSE,  and 
ISO  (with  regard  to  12207  and  15504).  The  goals  for  such  guidance  would  be  not  only  the 
publication  of  implementation  guides  and/or  white  papers,  but  also  providing  a  forum  for 
discussion  of  submitted  questions.  The  objective  is  to  move  beyond  the  limitations  of  “de¬ 
scriptive”  models  into  specific  “prescriptive”  suggestions,  “how  to”  guidance,  and  detailed 
examples  of  typical  implementations.  A  key  critical  success  factor  for  any  such  guidance  is 
that  the  organizations  performing  process  improvement  and  seeking  appraisals  see  the  same 
guidance  as  the  Lead  Assessor  and  Lead  Evaluator  groups.  The  working  group  felt  that  pro¬ 
viding  this  guidance  would  not  only  promote  greater  consistency  and  reliability  in  high  ma¬ 
turity  appraisals,  but  it  would  help  move  organizations  away  from  the  self-defeating  perspec¬ 
tive  of  “what  is  the  least  I  can  get  away  with  and  still  pass.” 

5.2.5  Achievement  of  Greater  Reliability  for  High  Maturity 
Appraisals 

The  working  group  chose  once  again  to  address  this  last  topic  in  a  free-form  manner. 

The  primary  question  underlying  this  third  topic  is:  What  steps  can  be  taken  to  review  and 
improve  the  reliability  of  high  maturity  appraisals?  The  working  group  further  refined  this 
broad  question  into  four  specific  topic  areas: 

1 .  Increased  focus  on  business  results 

2.  Analysis  of  appraisal  data  received  by  the  SEI 

3.  Quality  assurance  “audits”  of  appraisals  and  appraisers 

4.  Guidelines  for  reassessment  of  Level  5  rated  organizations 

Regarding  the  topic  of  increased  focus  on  business  results,  the  working  group  members  made 
the  following  points: 
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•  Now  that  appraisals  are  starting  to  use  Confidence  Reports,  one  possible  component  of  a 
Confidence  Report  for  high  maturity  ratings  would  be  data  on  organizational  perform¬ 
ance  that  correlates  with  the  expectation  of  high  maturity.  (This  is  similar  to  the  idea  of 
using  behavioral  maturity  level  indicators  in  Confidence  Reports,  suggested  in  the  SEPG 
2001  conference  paper  by  Judah  Mogilensky.) 

•  The  working  group  wanted  to  emphasize  that  there  was  no  suggestion  here  to  change  the 
formal  maturity  rating  approach  to  include  business  results  as  a  factor. 

•  It  was  also  noted  that  business  results  could  be  captured  in  the  Organizational  Question¬ 
naire  submitted  to  the  SEI,  so  that  data  on  the  correlation  between  high  maturity  levels 
and  improved  business  performance  could  be  analyzed. 


Regarding  the  topic  of  analysis  of  appraisal  data,  the  working  group  members  made  the  fol¬ 
lowing  points: 

•  The  existing  findings  in  the  Assessment  Findings  Briefings  and  Final  Reports  from  high 
maturity  assessments  in  the  SEI  database  could  be  used  to  establish  a  baseline  to  support 
semantic  and  content  analysis  of  such  findings. 

•  This  analysis  of  findings  could  be  augmented  in  the  future  by  having  the  appraisal  team 
submit  their  observation  databases  to  the  SEI,  and  extending  the  semantic  and  content 
analysis  to  these  observations.  Having  teams  submit  observations  will  require  specifying 
data  format  standards  and  sanitization  guidelines  to  a  significantly  greater  degree  than 
what  is  needed  now  for  Findings  Briefings  and  Final  Reports. 

•  Once  the  baseline  is  established,  the  SEI  could  perform  an  ongoing  review  of  appraisal 
results  to  identify  trends  and  patterns  in  the  content  of  findings  and  observations.  These 
trends  and  patterns  can,  in  turn,  guide  improvements  to  the  appraisal  program  (in  terms  of 
method  documentation  and  method  training),  as  well  as  the  future  public  guidance  dis¬ 
cussed  in  the  previous  section. 


Regarding  the  topic  of  quality  assurance  audits  of  appraisals  (and  possibly  appraisers),  the 

working  group  members  made  the  following  points: 

•  It  would  be  a  helpful  start  for  the  SEI,  perhaps  in  conjunction  with  the  Lead  Assessor  / 
Lead  Evaluator  communities,  to  identify  a  common  set  of  measures  that  indicate  and 
characterize  appraisal  performance. 

•  These  measures  of  appraisal  performance  could  then  be  communicated  to  the  appraiser 
communities,  and  to  the  organizations  (high  maturity  and  others)  seeking  appraisals. 

•  It  would  also  be  helpful  for  the  SEI,  in  its  role  as  steward  of  the  models  and  the  appraisal 
methods,  to  define  and  implement  an  appraisal  quality  assurance  process  (moving  beyond 
the  current  candidate  appraiser  observation  reports  and  PAIS  submittals).  This  quality  as¬ 
surance  process  may  well  require  separate  components  to  address  the  separate  topics  of 
appropriate  method  implementation  and  lead  appraiser  performance  on  the  one  hand,  and 
knowledge  and  interpretation  of  high  maturity  model  practices  on  the  other  hand. 


Finally,  regarding  the  topic  of  guidelines  for  the  reassessment  of  Level  5  organizations,  the 
working  group  members  made  the  following  points: 
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•  In  the  absence  of  the  conventional  motivation  for  conducting  reassessments  (namely,  the 
desire  to  demonstrate  achievement  of  the  next  higher  maturity  level),  Level  5  organiza¬ 
tions  must  decide  for  themselves  when  a  reassessment  is  warranted,  considering  such  fac¬ 
tors  as: 

-  time  elapsed  since  the  last  assessment 

-  typical  project  life-cycle  duration 

-  documented  improvement  activities  and  results 

-  changes  in  the  organization’s  structure,  leadership,  ownership,  areas  of  business  fo¬ 
cus,  staff  turnover,  applicability  of  models,  etc. 

•  The  working  group  declined  to  suggest  any  fixed  formula  for  “required”  reassessments  of 
Level  5  organizations,  just  as  there  are  no  fixed  requirements  at  any  other  level  for  ex¬ 
actly  when  reassessments  are  to  be  conducted.  (The  conduct  of  periodic  assessments  is 
addressed  in  Organization  Process  Focus,  but  no  specific  reassessment  interval  is  speci¬ 
fied.) 

•  At  the  same  time,  it  was  pointed  out  that  the  buyers  of  the  services  of  organizations  rated 
at  Level  5  (or,  again,  at  any  other  level)  still  need  to  look  at  such  factors  as: 

-  Model  used,  method  used,  scope,  and  leader(s)  of  prior  appraisals 

-  Measures  of  organizational  process  improvement  since  the  last  appraisal 

-  Other  differentiators  of  the  organization  relative  to  its  competitors  (that  is,  other  dif¬ 
ferentiators  besides  maturity  level  rating  at  some  point  in  time) 

The  members  of  the  working  group  on  High  Maturity  Appraisals  enjoyed  their  task,  and  felt 
that  they  had  been  able  to  have  useful  discussions  of  the  topics  summarized  in  this  report. 

5.3  Working  Group  1.3-  Six  Sigma 

The  participants  in  this  working  group  included  Bill  Curtis,  Bob  Hoekstra,  Asha  Goyal,  An¬ 
thony  D’Souza,  Mary  Lynn  Penn,  Joan  Romaine,  Kelley  Butler,  Jeannine  Siviy,  Christian 
Hertneck,  Jim  McHale,  and  Anita  Carleton. 

The  working  group  leader  was  Kelley  Butler.  The  presenter  was  Bill  Curtis.  The  facilitator 
was  Jeannine  Sivy.  The  recorder  was  Christian  Hertneck. 

5.3.1  Hypotheses  and  Observations 

The  questions  that  inspired  the  creation  of  this  working  group  included: 

•  What  does  Six  Sigma  really  mean  /  imply? 

•  Compare  /  contrast  Six  Sigma  and  the  Software  CMM 

•  How  to  sell  Six  Sigma  to  a  software  organization  and  when  /  how  to  start  Six  Sigma 


During  initiation  of  the  working  group,  the  following  questions  or  observations  were  posed 
by  the  working  group  members: 

•  Six  Sigma  as  a  toolbox,  a  collection  of  practices  for  continuous  improvement 
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•  Six  Sigma  as  a  method  for  driving  the  ownership  of  quality  to  the  project  groups/working 
level 

•  Six  Sigma  as  a  framework  for  improvement  beyond  Level  4  and  Level  5.  as  a  method  to 
“re-energize”  the  improvement  efforts 

The  issues  that  the  working  group  decided  to  discuss  in  more  detail  were: 

•  How  to  start  Six  Sigma 

•  Software  CMM,  People  CMM,  and  Six  Sigma 

•  High  Maturity  Organizations  and  Six  Sigma 

5.3.2  How  to  Start  Six  Sigma 

It  is  important  to  understand  that  Six  Sigma  does  not  require  a  maturity  level  and  may  be 
started  at  any  time,  but  it  must  have  management  support.  Training  is  key  and  it  is  best  to 
start  with  one  small  project  and  with  simple  problems  and  data  so  that  a  business  case  can  be 
made.  The  change  agents  “own”  the  Six  Sigma  process  and  Six  Sigma  involves  the  entire 
organization,  not  just  software. 

Being  at  a  higher  level  of  maturity  aids  in  the  implementation  of  Six  Sigma  due  to  the  avail¬ 
ability  of  data  and  the  focus  on  continuous  improvement  and  process  changes.  Additionally, 
Six  Sigma  focuses  on  the  problems  that  are  most  important  to  an  organization  and  its  bottom 
line,  e.g.,  shifting  the  discovery  of  defects  to  earlier  in  the  process. 

5.3.3  Software  CMM,  People  CMM,  and  Six  Sigma 

While  the  Software  CMM  focuses  on  transformation  of  the  organization  and  doesn’t  explic¬ 
itly  require  better  results.  Six  Sigma  drives  deeper  into  the  process  and  requires  measurable 
results.  Six  Sigma  also  enforces  many  of  the  principles  of  Personal  Software  Process  (PSP) 
and  Team  Software  Process  (TSP)  along  with  ISO  9001:20(X). 

Six  Sigma  clarifies  the  Level  4  and  5  goals  and  gives  tools  for  analyzing  the  process  accord¬ 
ing  to  business  goals.  Level  2  of  the  Software  CMM  is  about  local  learning  that  fits  well  with 
using  Six  Sigma  on  individual  projects.  Software  CMM  Level  3  begins  process  mapping  and 
actual  data  analysis  begins  at  Level  4.  The  original  basis  for  Software  CMM  Level  5  is  the 
beginning  of  data-driven  improvement,  the  foundation  of  Six  Sigma. 

Six  Sigma  starts  within  the  organization  with  a  strong  foundation  in  training,  focusing  of  de¬ 
velopment  of  the  workforce  and  ownership  of  the  process,  much  like  the  focus  of  the  People 
CMM. 
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5.3.4  High  Maturity  Organizations  and  Six  Sigma 

Software  CMM  Level  5  implies  that  change  is  built  into  the  process,  Six  Sigma  focuses  on 
the  creation  of  the  change  engine  -  giving  continuous  and  measurable  improvement  and  help¬ 
ing  to  ensure  that  the  organization  is  collecting  the  “right”  data.  Six  Sigma  allows  a  Software 
CMM  Level  5  organization  to  move  beyond  the  reference  model  and  build  an  improvement 
foundation  based  upon  the  Six  Sigma  tools  and  techniques. 

5.3.5  Experiences  from  Boeing  and  Lockheed  Martin. 

Mary  Lynn  Penn  of  Lockheed  Martin  and  Joan  Romine  of  Boeing  shared  their  experiences 
with  Six  Sigma.  For  Boeing,  it  was  originally  applied  to  manufacturing  and  they  are  now 
bringing  it  to  software  with  a  focus  on  delivered  product  quality  and  reducing  the  cost  of 
quality.  One  of  their  first  efforts  was  to  shift  the  discovery  of  defects  to  earlier  in  the  process. 

Lockheed  Martin  gained  commitment  from  the  top  and  trained  the  senior  management  so  that 
they  understood  the  process.  Their  president  has  set  goals  for  the  process  owners  with  a  cur¬ 
rent  example  being  a  goal  of  a  30%  defect  discovery  shift.  Ms.  Penn  discussed  the  involve¬ 
ment  of  the  entire  workforce  and  the  fact  that  they  are  seeing  a  new  culture  emerge.  She  de¬ 
scribed  their  Process  Board  and  that  they  zoom  in  on  specific  parts  of  the  process.  She  also 
discussed  the  need  for  specific  data. 

5.3.6  Recommendations  for  High  Maturity  Organizations 

As  a  result  of  the  working  group’s  discussions,  the  following  recommendations  for  organiza¬ 
tion  seeking  to  benefit  from  Six  Sigma  were  formulated: 

•  Gain  top-level  management  support.  Show  how  Six  Sigma  can  be  used  as  tool  and  the 
“next  step”  after  reaching  Software  CMM  Level  5.  Educate  the  entire  organization  on  the 
different  models,  methods,  and  tools. 

•  Use  Six  Sigma  to  increase  practitioner  ownership,  to  give  the  practitioners  a  new  tool,  to 
generate  new  interest,  to  increase  process  agility. 

•  When  implementing  Six  Sigma,  place  focus  on  training  and  piloting  with  one  project  and 
one  problem. 

•  And,  if  currently  using  Six  Sigma,  publish  experiences  and  results. 

5.3.7  Recommendations  for  the  SEI 

As  a  result  of  the  working  group’s  discussions,  the  following  recommendations  for  the  SEI 
were  formulated: 

•  Retain  the  Software  CMM  and  the  focus  it  brings  to  software. 

•  Explore  including  Six  Sigma  in  future  versions  of  the  Software  CMM  as  a  tool  to  aid 
Level  4  and  Level  5  organizations  in  continuous  and  measurable  improvement.  The  very 
rich  SPC  tool  kit  provided  by  Six  Sigma  could  make  Software  CMM  Level  4  and  Level  5 
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implementation  more  focused  and  less  theoretical.  Today  Level  4  recommends  process 
but  is  not  supported  by  a  toolkit. 

•  Recognize  that  Six  Sigma  is  not  standardized,  that  it  has  a  wide  spectrum  of  tools.  Focus 
on  referring  the  relevant  tools  from  the  Six  Sigma  assortment 

•  Publicly  recognize  Six  Sigma  as  a  useful  tool  for  software  organizations.  Publish  articles 
in  publications  such  as  CrossTalk. 

•  Show  the  relation  between  the  different  models  and  frameworks.  Show  the  connections, 
mappings,  and  benefits.  Describe  the  broad  toolset  necessary  to  develop  a  sound  quality 
management  system. 

•  Develop  SEI  courses  on  the  application  of  Six  Sigma  to  software  organizations. 

5.4  Working  Group  1.5:  CMMI 

The  participants  in  this  working  group  included  Julie  Barnard  (United  Space  Alliance),  Bruce 
Boyd  (The  Boeing  Company),  Lynn  Carter  (SEI),  Mary  Beth  Chrissis  (SEI),  Suzie  Garcia 
(SEI),  Diane  Gibson  (SEI),  Vivek  Govilkar  (iFlex  Solutions),  Craig  Hollenbach  (Litton 
PRC),  Mike  Konrad  (SEI),  Gerry  Ourada  (Lockheed  Martin),  Lynn  Penn  (Lockheed  Martin), 
Lita  Schulte  (The  Boeing  Company),  Raj  Shekher  (Mastek),  Ashok  Sontakke  (Zensar  Tech¬ 
nologies),  and  Albert  Soule  (SEI). 

The  facilitator  was  Mike  Konrad.  The  scribe  was  Lynn  Penn.  The  recorder  was  Diane  Gib¬ 
son.  Julie  Barnard  and  Bruce  Boyd  volunteered  to  be  the  report  editors.  Julie  Barnard  pre¬ 
sented  the  working  group’s  finding  to  the  workshop. 

Note  that  there  were  two  separate  working  groups  on  the  CMMI  topic  at  the  workshop,  due 
to  the  level  of  interest  in  the  topic  expressed  by  the  workshop  attendees.  This  CMMI  working 
group  (1.5)  was  the  first  of  the  two  working  groups  convened  on  the  topic.  Different  partici¬ 
pants  were  involved  in  each  of  the  two  CMMI  working  groups. 

5.4.1  Hypotheses  and  Observations 

This  section  discusses  the  observations,  hypotheses,  and  propositions  that  initiated  the  dis¬ 
cussion.  Also  included  are  the  results  of  brainstorming  activities  that  did  not  become  a  work¬ 
ing  group  consensus. 

The  working  group  brainstormed  the  following  set  of  initial  questions  to  be  discussed; 

•  How  does  an  existing  high  maturity  software  organization  integrate  with  a  relatively  im¬ 
mature  systems  engineering  organization  when  transitioning  from  Software  CMM  to 
CMMI? 

•  What  is  the  next  step  for  CMMI  development  and  release?  How  do  we  plan  for  CMMI 
over  the  next  two  years? 
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•  How  will  CMMI  help  organizations  that  develop  custom  software?  How  will  CMMI, 
which  covers  systems  engineering,  help  those  who  don’t  see  systems  engineering  as  part 
of  their  business?  How  practical  are  CMMI  assessments  if  they  can  take  2-3  weeks? 

•  Where  can  we  find  mapping  from  Software  CMM  to  CMMI  for  higher  maturity  level 
organizations? 

•  What  are  the  qualifications  for  SCAMPI  assessors?  How  long  do  assessments  take? 

•  How  do  you  integrate  a  number  of  separate  legacy  organizations  using  CMMI  following 
mergers?  How  difficult  is  it  to  cover  a  diverse  organization  with  a  common  CMMI  as¬ 
sessment? 

•  The  CMMI  product  development  team  is  interested  in  hearing  the  concerns  of  industry 
on  the  model.  How  do  you  apply  CMMI  to  commercial  products  not  currently  covered  by 
CMMI? 

•  When  merging  various  companies  into  one,  shouldn’t  we  deal  with  the  merger  issues 
first,  then  CMMI?  Software  is  just  a  part  of  what  the  company  does — CMMI  will  cause 
companies  to  pull  in  more  of  the  organization  than  just  engineering  into  their  improve¬ 
ment  plans — for  example,  things  in  the  factory,  or  in  the  quality  side  of  the  house.  We’re 
implementing  Level  4  in  software  for  commercial  production  because  we  know  it  is  the 
only  way  to  meet  the  contract.  We  would  like  to  do  similar  improvement  on  the  engineer¬ 
ing  side  of  the  house. 

•  We  are  currently  Level  5  with  both  Software  CMM  and  Systems  Engineering  CMM.  We 
are  currently  in  the  rollout  and  integration  of  CMMI.  We  were  also  member  of  the  CMMI 
working  group  previously.  We  want  to  see  how  CMMI  has  evolved  and  get  results  from 
the  pilot  assessments.  We’d  also  like  to  hear  interpretations  of  the  continuous  vs.  staged 
representations. 

•  SEI  would  like  to  understand  what  high  maturity  organizations  think  about  the  practices 
at  Levels  4  and  5  in  CMMI  Version  1.01.  How  different  are  they  from  Version  1.1  of 
Software  CMM?  How  much  is  enough  to  be  assessed  at  Levels  4  and  5?  How  much  of 
the  product  life  cycle  needs  to  be  brought  under  statistical  process  control? 

•  We  have  a  similar  question  regarding  Technology  Change  Management  -  When  high  ma¬ 
turity  organizations  evaluate  and  decide  to  adopt  new  technology,  is  that  activity  sup¬ 
posed  to  be  under  Statistical  Process  Control  (SPC)?  Since  technology  is  changing 
quickly  and  changes  are  happening  so  fast,  should  it  be  [under  SPC]? 

•  What  are  the  lessons  learned  that  could  be  used  by  a  novice  organization  applying  CMMI 
vs.  an  organization  experienced  with  Software  CMM? 

•  Where  is  CMMI  going  in  the  future?  Will  it  include  the  People  CMM?  Will  it  apply  to  an 
Information  Technology  organization?  Will  there  be  one  assessment  for  the  entire 
organization? 

•  What  experiences  have  people  had  with  CMMI  lessons  learned?  What  about  extending 
the  CMM  to  other  areas? 

•  How  do  you  perform  SPC  for  areas  other  than  software  specific  development? 


Several  working  group  members  asked  for  background  information  on  CMMI.  Mike  Konrad 
(SEI)  provided  a  brief  summary  of  relevant  CMMI  information  to  the  group.  Since  this  in- 
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formation  is  readily  available  from  the  SEI  Web  site  and  other  sources,  it  is  not  reproduced  in 
this  report.  Some  of  the  members  asked  about  mapping  of  various  other  models  and  standards 
to  CMMI.  It  was  noted  that  some  useful  mappings  are  available  on  the  Software  Technology 
Support  Center  (STSC)  Web  site:  <http://www.stsc.hill.af.mil/>.  Additionally,  the  SEI  Web 
site  includes  pointers  to  the  same  mapping  documents  resident  on  the  STSC  Web  site  that 
compare  the  Software  CMM  to  the  CMMI  and  vice-versa. 

During  initiation  of  the  working  group,  the  working  group  members  posed  the  following 
questions  or  observations: 

Observation  #1:  Are  software  organizations  using  CMMI? 

It  was  argued  that  software  engineering  and  system  engineering  are  not  really  separate  disci¬ 
plines  -  either  or  both  have  been  characterized  as  encompassing  the  other.  Engineering  proc¬ 
ess  areas  talk  about  defining  the  processes  that  go  with  developing  and  using  the  product: 
manufacturing,  customization,  training,  repair,  etc.  A  broader  view  of  life  cycle  is  required — 
for  example,  maintainers  of  products  who  know  all  the  effort  required  to  make  a  product  use¬ 
ful  and  keep  it  functional.  CMMI  gives  more  attention  to  these  stages.  Organizations  that  are 
only  developing  software  still  have  integration  issues  about  installation,  help  desk,  tech  sup¬ 
port,  etc.  CMMI  gives  software  organizations  that  develop  applications  a  model  to  include 
other  aspects  of  product  development;  e.g.,  relevant  stakeholders.  CMMI  practices  integrate 
more  decision  making  and  parts  of  product  development.  The  value  of  Software  CMM  was  to 
give  focus  to  neglected  areas  such  as  support  and  project  management.  CMMI  folds  in  les¬ 
sons  learned  from  Software  CMM  and  from  engineering  (EIA  731).  CMMI  tried  to  capture 
these  lessons  learned.  At  Software  CMM  maturity  levels  4  and  5  one  of  the  lessons  learned 
was  that  the  Technology  Change  Management  and  Process  Change  Management  Key  Process 
Areas  could  be  merged  (from  the  workshop  on  TCM);  also,  product  lines  were  included  in 
Software  CMM  Version  2.  There  is  still  an  opportunity  to  look  upstream,  downstream,  and 
laterally  for  information  about  the  products  and  services  an  organization  provides. 

The  issue  that  the  working  group  discussed  in  more  detail  was: 

Marketing 

•  Commercial  organizations  have  different  issues  from  defense  contractors;  for  example, 
marketing  is  so  much  more  important.  Does  CMMI  focus  more  on  marketing,  or  might 
it? 

•  System  engineering  issues — customer  requirements,  product  management,  etc.  -  often 
focus  on  information  needed  by  marketing.  Someone  is  working  on  a  Masters  thesis  on  a 
CMM  for  marketing. 

•  Does  the  model  sub-optimize  the  commercial,  marketing  approach? 

•  CMU  is  working  with  private,  commercial  companies — using  the  CMM  as  a  focal  point 
(e.g..  Sun,  Adobe,  3Com,  Oracle). 
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•  Look  at  the  participants  in  the  development  of  CMMI — most  were  defense  contractors — 
but  there  were  some  commercial  companies  (Motorola,  Ericsson);  all  were  either  defense 
or  telecommunications  companies.  Software  CMM  began  as  a  tool  for  defense  contrac¬ 
tors  with  the  SEI  shepherding  the  flock.  The  SEI  wants  input  from  commercial  organiza¬ 
tions,  but  Software  CMM  is  something  for  software  development — marketing  folks  are 
not  very  excited  about  it.  It  would  be  better  if  there  were  something  that  comes  from 
marketing  to  get  them  involved.  CMMI  has  stakeholder  involvement;  i.e.,  coordination 
with  the  stakeholders  group;  but  it  suffers  from  not  having  an  explicit  “marketing  the 
product”  process  area.  Maybe  this  will  happen  in  the  future? 

•  Business  acquisition  is  a  major  focus  of  companies.  Requirements  are  important,  but  also 
need  the  business  acquisition  process  (business  development,  marketing)  to  focus  on 
risks,  etc.  CMMs  give  a  push  to  engineering  but  not  to  marketing. 

•  When  we  bring  software  or  engineering  improvement  to  the  boardroom,  CMM  seems 
parochial.  They  are  more  interested  in  growing  the  business  and  making  profits.  We  need 
to  bring  software  improvement  back  into  improving  the  business  and  addressing  business 
issues. 

•  People  are  making  the  argument  that  CMMI  has  to  be  translated  for  specific  business 
contexts.  If  CMMI  constrains  the  ability  to  meet  business  goals,  we  would  like  to  hear 
more  examples  or  evidence  of  this. 

•  CMMI  has  weaknesses  in  the  areas  of  marketing  and  business  development.  Whatever 
the  SEI  wants  to  address  regarding  marketing  needs  to  be  clearly  addressed  in  training  for 
instructors  and  for  assessors.  We  don’t  expect  this  to  really  change  in  next  three  years. 

•  Is  there  a  set  of  principles  that  can  be  used  as  guidelines  in  other  parts  of  organization 
that  are  outside  the  CMMI?  For  example,  are  there  architectural  principles  for  adding 
new  disciplines,  or  new  generic  practices?  Is  there  a  single  model  for  product  develop¬ 
ment  processes  /  organizations  and  a  path  for  adding  other  disciplines  and  application  en¬ 
vironments? 

•  We  are  looking  to  members  of  this  group  for  people  who  are  applying  CMMI  in  the 
commercial  world  and  in  other  areas  for  their  insights. 


Observation  /  Question  #2;  Standard  CMMI  Assessment  Method  for  Process  Improve¬ 
ment  (SCAMPI)  issues 

•  Initial  concern — SCAMPI  takes  such  a  long  time  and  is  an  intense  effort.  Today  it  takes 
about  100  hours  (clock  time)  over  nine  days.  The  second  time  a  Lead  Assessor  takes  less 
time.  It’s  getting  more  like  CBA IPI,  plus  there  are  some  innovations  that  could  also  be 
used  in  CBA  IPI. 

•  Concern  with  CMMI — It  is  possible  (likely?)  that  the  requirement  for  Software  CMM 
Level  3  will  be  changed  to  CMMI  Level  3  or  something  equivalent.  How  assessments  are 
done  is  critical  here. 

Observation  /  Question  #3:  CMMI  changes 

•  CMMI  Version  1. 1  is  expected  to  be  released  in  December,  2001  and  then  be  stabilized 
for  four  or  more  years.  The  desire  is  that  vl.l  will  be  similar  enough  to  v  1.0  that  folks 
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won’t  need  to  be  retrained.  No  changes  are  expected  in  a  number  of  process  areas;  goals 
and  practices  should  be  pretty  much  the  same. 

•  The  biggest  changes  will  be  in  the  evaluation  techniques  (SCAMPI).  From  OSD — there 
will  be  a  “source  capability  evaluation”  method.  We  do  anticipate  changes  in  the  assess¬ 
ment  method  to  save  time  and  take  advantage  of  lessons  learned  in  pilot  assessments. 

•  Regarding  which  new  disciplines  will  be  added  to  the  model:  there  is  no  clear  direction. 
We  will  add  acquisition  in  some  form;  security  is  pushing  also;  enterprise  modeling,  pro¬ 
gram  office,  etc.  are  all  being  raised  but  no  decision  has  been  made.  People  CMM  Ver¬ 
sion  2  is  being  crafted  to  be  compatible  with  CMMI. 

Observation  /  Question  #4:  Expansion  to  other  areas 

•  One  company  tried  to  include  other  disciplines  using  the  Software  CMM — but  when  sen¬ 
ior  management  heard  about  CMMI  they  stopped  that  initiative.  They  were  moving  im¬ 
provement  into  product  development  processes,  which  is  more  than  just  integrated  engi¬ 
neering  processes. 


Observation  /  Question  #5:  SPC 

•  CMMI  promotes  doing  SPC  where  the  business  case  suggests  you  need  SPC.  If  TCM  is 
very  important  to  an  organization,  then  they  might  want  to  use  SPC  for  TCM.  In  other 
organizations,  this  may  not  be  needed. 

•  How  much  SPC  is  enough?  Are  there  any  universal  processes  that  should  always  be 
placed  under  SPC? 


5.4.2  Novice  vs.  Mature  Organizations 

How  will  CMMI  apply  to  a  novice  organization  vs.  one  with  a  mature  software  organization? 

If  you  have  several  pockets  of  high  maturity  practices,  how  do  you  apply  CMMI  to  additional 

areas?  Examples  of  high  maturity  organization  experiences: 

•  System  engineers  wanted  to  learn  and  adopt  processes  used  by  software  folks  when  they 
participated  in  Integrated  Product  Teams  (IPTs)  with  Level  5  software  engineers. 

•  An  executive  commented  that  their  company  no  longer  received  complaints  about  soft¬ 
ware,  but  they  still  got  complaints  about  other  engineering  domains,  so  they  tried  to  ap¬ 
ply  Software  CMM  principles  to  top  level  product  development  processes,  including  en¬ 
gineering,  business,  and  end-to-end  processes.  They  intend  to  apply  CMMI  at  some 
point.  This  was  an  example  of  executive  push.  Having  demonstrated  the  benefits  of 
CMM  in  software,  they  wanted  to  apply  it  more  broadly.  It  was  somewhat  awkward  to 
apply  CMM  outside  of  software,  but  it  was  a  valuable  exercise.  There  was  previously  no 
concept  of  peer  reviews  of  business  plans,  but  the  practice  was  introduced  because  it 
made  sense  -  they  were  important  documents. 

•  Experience  getting  ready  for  a  CMMI  pilot:  It  was  recognized  that  Software  CMM  had 
added  value  and  been  beneficial  to  the  software  community,  so  engineering  management 
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was  ready  to  try  it,  even  though  they  didn’t  know  exactly  what  it  was.  They  saw  real 
benefits  from  the  Level  3  aspects,  rather  than  the  high  maturity  aspects. 

•  Another  organization  doesn’t  have  mature  system  engineering  and  has  software  engineer¬ 
ing  communities  that  are  diverse.  There  are  up-front  components  and  back-end  compo¬ 
nents  that  haven’t  paid  enough  attention  to  process  improvement.  When  Systems  Engi¬ 
neering  CMM  came  out,  they  looked  for  areas  to  piggyback  usage  between  software  and 
system  engineering — where  they  could  use  software  processes  in  the  system  engineering 
world.  System  engineering  processes  became  part  of  the  improvement  structure.  Total 
Quality  Management  (TQM)  also  provided  some  foundational  principles  that  extended 
across  disciplines — TQM  was  highly  leveragable.  Quality  techniques  of  the  TQM  ap¬ 
proach  were  applied  across  the  organization  first,  then  they  applied  Software  CMM. 

•  Another  organization  matured  in  both  system  engineering  and  Software  CMM.  They  re¬ 
placed  SEPGs  with  an  Engineering  Process  Group,  with  members  from  software,  system 
engineering,  quality,  configuration  management,  business  development,  process  im¬ 
provement,  etc.  They  needed  this  joint  structure  to  achieve  joint  improvement.  The  result¬ 
ing  processes  are  credible  because  they  have  a  representative  from  each  area  in  the  group. 
When  they  establish  processes  in  software,  they  need  involvement  of  estimators,  manag¬ 
ers,  etc.  It  is  beneficial  to  put  all  groups  together  in  the  process  group  so  they  define 
processes  together,  and  allow  for  a  more  natural  progression.  This  was  the  biggest  bene¬ 
fit:  to  have  everyone  work  together.  As  they  pulled  new  organizations  into  the  company 
(by  merger),  the  process  group  was  able  to  deal  with  the  mapping  and  the  processes  in 
the  new  organization. 


Question:  If  an  organization  is  starting  from  scratch  (no  previous  CMM  experience),  do  you 
recommend  first  focusing  on  software  and  then  including  other  engineering  areas?  Or  should 
you  go  for  them  all  concurrently? 

•  One  organization  embarked  on  an  enterprise  level  improvement  effort  by  first  getting 
software  to  Level  3.  They  wished  they  had  done  it  all  together  from  the  start.  Software 
had  a  lot  invested  in  their  approach,  and  had  to  convince  others  that  their  way  was  good 
for  all. 

Conclusion:  Depending  upon  the  organizational  circumstances  there  were  examples  de¬ 
scribed  that  support  both  positions  -  software  can  be  an  inspiration  for  other  parts  of  the  or¬ 
ganization,  or  it  can  be  best  to  introduce  change  across  the  entire  engineering  organization  at 
one  time. 

Another  organization  implemented  ISO  9001  first,  then  Software  CMM,  then  TQM  in  1995. 
They  included  Human  Resources  and  other  areas,  including  marketing,  into  their  TQM  im¬ 
plementation.  They  will  limit  the  use  of  CMMI  while  still  looking  at  the  enterprise  through 
TQM  and  benchmarking. 

CMMI  reinforces  the  shared  model — defining  high  leverage  process  areas  for  all  of  the  or¬ 
ganization.  It  highlights  commonality  and  areas  for  integration. 
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If  an  organization  is  looking  at  leveraging  high  maturity  experience  to  areas  with  no  maturity, 
the  CMMI  Generic  Practices  (GPs)  are  the  basic  behavioral  principles  that  can  be  used  in  any 
discipline.  They  become  a  model  to  use  in  every  discipline.  The  set  of  GPs  is  a  candidate  for 
high  leverage  processes  for  different  areas.  Generic  practices  can  be  used  in  many  areas  for 
process  improvement. 

5.4.3  Impact  of  Organizational  Mergers 

What  is  the  effect  of  organizational  mergers  on  high  process  maturity?  Some  examples  fol¬ 
low  of  high  maturity  organization  experiences  with  acquisitions,  mergers,  and  reorganizations 
and  its  associated  effect  on  blending  processes  and  process  maturity: 

•  One  large  company  of  approximately  8500  people  integrated  another  organization  of  ap¬ 
proximately  2000  people  together  through  a  merger.  In  discussions  on  the  process  stan¬ 
dard  to  be  used  for  the  newly  formed  organization  of  10,000  people,  the  company  level 
process  group  began  to  talk  about  what  level  of  process  commonality  should  exist  across 
the  organization.  The  standard  process  existed  prior  to  the  merger;  however,  it  was  rec¬ 
ognized  that  the  existing  process  standard  might  not  be  immediately  achievable  by  the 
new  parts  of  the  organization.  Representatives  from  the  new  part  of  the  organization  par¬ 
ticipated  in  the  company  process  group  to  review  the  process  standard.  The  company 
process  standard,  which  is  the  set  of  minimum  standard  processes  to  be  used  for  all  or¬ 
ganization,  covers  20  processes.  Each  process  is  represented  in  about  1  page  of  structured 
English  text  and  is  task  oriented.  As  a  result  of  the  merger,  the  process  standard  was 
modified  so  that  the  level  of  detail  was  raised  for  the  newly  formed  organization  to  some¬ 
thing  that  everyone  could  live  with,  that  their  supporting  procedures  could  support,  and 
that  everyone  across  the  organization  could  use  the  process  standard.  As  the  revised 
higher-level  standard  was  adopted  and  deployed  by  the  newer  group,  then  the  company 
process  group  could  revisit  the  level  of  detail  in  the  standard.  The  process  standard  began 
to  get  more  detailed  as  more  parts  of  the  organization  used  similar  processes.  At  a  later 
point  in  time  when  a  second  new  group  was  incorporated  into  the  company,  the  review  of 
the  processes  began  again  to  determine  what  lowest  common  denominator  of  standard 
process  could  be  accepted  across  the  board.  The  detailed  process  information  was  cap¬ 
tured  and  retained  during  these  revision  periods  so  that  the  processes  could  be  tailored 
subsequently  and  include  the  details  as  appropriate. 

•  Another  organization  of  about  2200  software  people  tried  a  similar  approach  when  af¬ 
fected  by  a  merger.  One  of  their  big  struggles  was  with  the  customer  Defense  Contract 
Management  Agents,  who  report  to  different  program  offices.  The  program  offices  were 
resisting  the  standardization  of  processes,  because  they  are  site  focused.  They  are  com¬ 
fortable  with  the  way  things  are  and  don’t  want  to  see  changes.  In  this  case,  if  the  cus¬ 
tomer  was  allowed  to  drive  the  standardization,  then  there  could  be  backward,  instead  of 
forward  progress. 

In  addition  to  the  software  process  impacts  associated  with  mergers,  there  was  discussion  of 
impacts  to  areas  such  as  Human  Resources,  marketing,  financial  practices  and  the  importance 
of  these  issues.  In  one  company,  there  was  focus  on  workforce  issues  (e.g.,  through  the  Peo¬ 
ple  CMM)  once  the  high  level  process  group  was  established  and  the  technical  processes  had 
achieved  a  high  level  of  maturity. 
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In  one  organization,  the  affected  groups  had  to  evaluate  their  compliance  to  the  standard  pro¬ 
cess  through  their  implementation  of  it.  This  included  use  of  tools  and  detailed  procedures  in 
implementation  of  the  standard  process.  For  example,  the  configuration  management  process 
standard  contained  a  list  of  required  tasks.  Different  parts  of  the  organization  used  different 
configuration  management  tools;  however,  as  long  as  the  tools  accomplished  the  required 
tasks  and  roles  in  the  high  level  processes,  then  there  was  no  need  to  change  tools.  If  the  tools 
in  use  did  not  accomplish  the  required  configuration  management  tasks,  then  that  part  of  the 
organization  needed  to  show  how  it  would  accomplish  all  of  the  required  tasks  from  the  stan¬ 
dard.  In  some  cases  this  resulted  in  a  change  of  tool  use.  New  projects  were  expected  to  use 
centrally  supported  tools.  A  similar  approach  was  used  in  evaluating  the  compliance  of  low- 
level  procedures  and  their  support  of  the  high  level  process.  If  part  of  the  organization  used  a 
procedure  that  accomplished  all  the  required  standard  tasks,  then  it  could  be  maintained. 
Commonality  was  sought  where  it  made  sense.  For  example,  different  kinds  of  peer  reviews 
were  being  practiced  in  the  organization.  To  try  and  standardize  the  inspection  process,  a 
formal  kaizen  event  was  conducted  on  inspections.  This  resulted  in  the  same  form  of  inspec¬ 
tions  being  adopted  throughout  the  organization. 

In  another  organization,  mapping  and  standards  were  in  place  across  the  new  organization, 
but  training  and  implementation  were  lagging  behind  a  bit.  In  addition,  there  was  some  resis¬ 
tance  to  things  such  as  quantitative  management.  So,  that  part  of  the  organization  does  not 
achieve  maturity  level  4  in  the  targeted  time  period  (e.g.,  6  months.) 

One  organization  divests  itself  of  part  of  its  group.  The  Level  4/5  group  ended  up  trying  to 
maintain  their  maturity  level  through  “tribal  knowledge.”  They  winnowed  down  their  docu¬ 
mentation,  but  found  that  they  did  not  have  enough  detail  to  adequately  train  new  people.  The 
organization’s  veterans  knew  the  process,  but  many  of  them  were  retiring.  This  caused  a  need 
to  re-document  their  processes.  They  struggled  through  a  Level  3  assessment — ^and  are  climb¬ 
ing  back  up.  The  documentation  was  not  adequate  to  sustain  Level  4  when  they  lost  so  many 
people  who  had  institutionalized  processes  and  had  created  a  stable  organization.  This  or¬ 
ganization  recommended  that  documentation  be  evaluated  using  criteria  of  how  easily  some¬ 
one  can  pick  it  up  and  learn  to  do  the  processes. 

One  organization  represented  in  the  working  group  was  about  to  be  involved  in  a  merger  and 
was  seeking  suggestions  from  experienced  organizations  for  what  it  takes  to  maintain  process 
maturity  during  take-over.  Some  ideas  provided  from  “merger-experienced”  organizations 
were  provided; 

•  Be  careful  about  how  senior  management  describes,  documents,  and  represents  the  take¬ 
over.  In  one  case,  the  combining  of  organizations  was  declared  to  be  a  merger  and  not  to 
be  a  take-over  and  that  the  best  was  to  be  combined  from  each  of  the  merged  organiza¬ 
tions.  However,  the  management  of  the  combined  organization  was  all  from  the  organiza¬ 
tion  that  initiated  the  merger;  which  reflected  the  perception  of  take-over  rather  than 
merger.  Teams  need  to  be  established  as  soon  as  possible  and  should  begin  talking  before 
the  “thou  shall  do...”  is  issued  from  the  top  management. 
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•  In  another  organization,  a  lot  of  time  was  spent  getting  to  know  other  process  people  in 
the  organization  to  establish  contacts  and  exchange  process  ideas.  This  resulted  in  not  ar¬ 
tificially  forcing  any  combinations. 

•  In  another  instance,  a  Level  5  organization  merged  into  a  Level  3  organization.  New  peo¬ 
ple  began  to  work  with  and  help  the  Level  3  folks  with  their  issues.  They  worked  toward 
renaming  processes  -  no  one  retained  the  old  process  names,  but  rather  worked  toward 
creating  some  new  names  that  both  organizations  could  live  with  and  did  not  convey  any 
ties  to  the  past  organizational  structures. 

•  It  was  suggested  to  be  cautious  of  the  snob  factor  and  not  convey  one  organization  as 
“better”  than  the  other  even  when  there  are  differing  maturity  levels.  Ultimately,  there 
should  be  common  objectives  and  goals  that  unite  the  groups  and  the  “more  mature” 
groups  should  assist  the  “lesser  mature”  groups  and  do  so  in  humility. 

•  In  order  to  posture  a  high  maturity  company  so  that  the  impact  of  mergers  is  lessened,  it 
was  suggested  that  metrics  be  shared  with  new  organization  and  new  management  as  a 
part  of  the  familiarization  and  transition. 

5.4.4  Software  CMM  and  CM  Ml — Key  Differences? 

One  of  the  new  process  areas  in  CMMI  is  Measurement  and  Analysis.  With  the  Software 
CMM,  it  was  believed  that  there  wasn’t  enough  measurement  represented  at  lower  levels. 
Even  though  measures  existed  for  each  key  process  area,  they  were  often  considered  not  to  be 
useful  measures.  For  some  organizations  that  were  striving  toward  Level  4  maturity,  they 
sometimes  had  to  rethink  a  lot  of  measures  that  were  put  in  place  at  lower  level  key  process 
areas.  Some  organizations  reported  that  they  did  not  wait  until  maturity  levels  3  or  4  to  do 
process  measurement  and  quantitative  management,  but  rather  that  measurement  was  impor¬ 
tant  to  establish  at  lower  levels.  The  existence  of  Measurement  and  Analysis  is  one  of  the 
things  in  CMMI  that  may  help  organizations  get  to  higher  maturity  levels  faster  by  providing 
the  necessary  foundation. 

Some  lower  maturity  level  organizations  may  not  see  the  need  for  metrics  because  they  are  so 
busy  just  trying  to  get  the  basics  done.  Projects  may  be  producing  measurable  data,  but  until 
the  Level  4  processes  and  improvement  methods  are  established,  they  may  not  see  any  bene¬ 
fits. 

One  organization  reported  that  they  performed  measurement  for  a  long  time.  However,  in 
1989  they  received  a  letter  demanding  improvement,  because  their  costs  were  too  high, 
schedules  were  unpredictable,  and  quality  was  poor.  They  performed  an  analysis  of  produc¬ 
ing  software  from  the  perspective  of  cost  and  schedule — that  was  better  than  focusing  on 
quality  (alone).  They  established  a  solid  earned  value  management  process.  What  did  they 
see  in  the  Software  CMM  at  that  time?  They  were  already  doing  measurement,  so  they  fo¬ 
cused  on  quality  and  configuration  management  because  that  was  new.  CMMI  is  saying  that 
measurement  is  important  at  lower  levels.  Organization  probably  shouldn’t  try  to  focus  on 
quality  alone,  but  rather  should  focus  on  cost  and  schedule  type  issues  as  well. 
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Another  area  of  key  change  in  CMMI  from  Software  CMM  is  the  level  of  detail  of  the 
engineering  processes  for  product  development.  The  questions  were  raised; 

•  Is  that  detail  helpful  for  process  improvement  or  is  it  a  hindrance? 

•  Is  the  lack  of  focus  on  software  a  problem  or  a  benefit? 

Engineering  processes  can  be  used  to  demonstrate  what  might  be  needed  and  where  one 
might  begin  in  implementing  CMMI.  The  CMMI  represents  the  basics  of  engineering  proc¬ 
esses.  However,  there  may  be  problems  encountered  during  implementation  for  Information 
Technology  organizations,  dot-coms,  and/or  shrink-wrap  organizations,  depending  on  previ¬ 
ous  experience,  types  of  software  development,  the  criticality  of  the  software  and  experience 
with  standards  in  the  past.  An  example  was  offered  of  a  backend  Web  company  with  no  exist¬ 
ing  life  cycle,  no  sense  of  process  or  project  management  or  defect  tracking  yet  CMMs  could 
help  the  start-ups  if  used  well. 

The  CMMI  is  not  just  a  set  of  processes,  but  a  model  or  a  guide  to  improve  processes.  It  ap¬ 
pears  some  are  using  the  CMMI  as  a  model  for  their  processes  rather  than  as  a  model  for 
measuring  “maturity”  of  processes.  Using  CMMI  as  a  model  for  measuring  processes  is  a 
viewpoint  from  a  mature  organization.  So,  what  can  a  mature  organization  do  with  CMMI? 
Do  they  have  to  rewrite  processes?  This  is  the  wrong  approach;  it  is  not  a  set  of  processes. 
When  a  mature  organization  looks  at  a  new  model  and  tries  to  learn  from  it,  somewhere  they 
need  to  ask  if  what  they  have  is  adequate  or  is  there  something  missing.  There  may  be  new 
insights  or  ideas  coming  from  a  new  model — and  once  you  know  the  new  idea,  you  want  to 
implement  it  and  gain  advantage  from  it.  Improvement  can  come  from  within  or  can  come 
from  outside  the  organization. 

Transition  to  CMMI  means  comparing  processes  against  a  new  model.  The  CMMI  embedded 
some  of  what  was  learned  about  maturity  in  organizations  (e.g.,  PCM,  TCM,  and  OLD).  Does 
CMMI  capture  the  paths  previously  taken  by  higher  maturity  organizations  better  than  the 
Software  CMM? 

5.4.5  Removing  Stovepipes  to  Implement  CMMI 

How  do  organizations  with  Software  CMM  experience,  but  no  CMM  experience  in  systems 
engineering,  remove  stovepipes  to  implement  CMMI?  One  Software  CMM  Level  4  organiza¬ 
tion  engaged  their  systems  engineering  people.  They  pulled  together  their  processes  to  be 
consistent  with  Level  3  software  processes  (on  one  particular  program).  Now,  some  system 
engineering  groups  exhibit  LI  or  L2  behavior,  and  now  they  have  to  deal  with  this.  There  are 
clearly  defined  interfaces,  project-by-project,  program-by-program.  Software  processes  are 
institutionalized,  but  interfaces  to  systems  engineering  are  chaotic. 

Another  organization  has  two  levels  of  system  engineering  (aircraft  level  and  detailed  system 
level).  Those  areas  that  are  associated  with  software  have  adopted  some  of  the  software  prac¬ 
tices.  This  hasn’t  been  transferred  to  the  aircraft-level  system  engineers.  The  software  organi- 
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zation  pulls  process  focused  behavior — and  the  spread  is  slow  and  resisted.  CMMI  can  bring 
such  organizations  to  the  awareness  of  process-focused  needs — especially  when  customers 
say  they  are  going  to  use  CMMI  to  evaluate  them. 

When  software  and  systems  are  tightly  coupled,  practices  do  diffuse — but  other  engineering 
areas  may  have  no  contact. 

In  one  organization,  the  software  process  owner  is  running  the  software  improvement  pro¬ 
gram  across  the  entire  organization.  The  engineering  process  improvement  effort  is  just  be¬ 
ginning  with  self-assessments,  documenting  processes,  and  evaluating  tools.  The  objective  is 
to  provide  measures  to  the  CEO  as  requested. 

Another  organization  described  a  leap-frogging  approach.  Software  engineering  was  way 
ahead  but  systems  engineering  was  working  with  the  software  processes.  When  the  company 
was  bought  out,  it  was  noted  that  one  of  major  problems  was  how  different  units  work  to¬ 
gether.  Systems  engineering  began  an  effort  to  document  processes  at  the  organization  level 
to  resolve  this  problem. 

Hypothesis:  A  major  difference  between  low  and  high  maturity  organizations  is  that  high  ma¬ 
turity  organizations  have  the  data  to  prove  and  demonstrate  that  their  improvements  are  suc¬ 
cessful. 

Since  there  is  no  mandate  to  use  CMMI,  one  reason  for  system  engineering  choosing  to  go 
ahead  with  CMMI  was  having  seen  the  success  of  software  engineering  using  Software 
CMM. 

In  another  case,  a  software  person  moved  over  to  systems  engineering  because  he  knew  the 
Software  CMM  and  improvement  methods  and  they  wanted  him  to  implement  the  systems 
engineering  processes.  In  another,  the  software  manager  was  made  equal  to  the  systems  engi¬ 
neering  manager,  where  previously  software  reported  to  systems. 

Organizations  choosing  CMMI  are  making  a  strategic  decision.  The  VP  of  Engineering  was 
the  sponsor  in  one  case.  CMMI  should  bring  another  organization  closer  to  looking  at  busi¬ 
ness  development  and  evaluation — so  sponsorship  may  be  at  a  higher  level. 

What  made  the  light  go  on  among  senior  executives  and  others?  In  one  case,  some  people 
(engineers  and  leaders)  recognized  problems  in  their  own  area  and  saw  what  was  happening 
with  process  improvements  elsewhere  in  the  organization — their  initiative  drove  a  bottom-up 
improvement  effort.  In  another  case,  an  enlightened  customer  made  a  huge  difference  by 
driving  the  organization  to  improvement  (e.g.,  the  customer  said  that  they  thought  it  would 
take  a  high  maturity  organization  to  win  the  contract).  Engineers  and  program  managers  have 
to  keep  reminding  senior  executives  of  customer  comments  in  order  to  maintain  support. 
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There  have  been  few  CMMI  assessments  at  this  point.  Many  organizations  are  now  looking 
at  CMMI,  making  improvements,  and  evaluating  internally  against  CMMI.  Some  organiza¬ 
tions  are  planning  for  formal  assessments  in  a  year  or  two. 

One  organization  is  performing  a  Pilot  Assessment.  They  formed  a  steering  group  at  the  be¬ 
ginning  of  the  year  to  plan  for  the  assessment.  They  have  had  Intro  to  CMMI  taught  on  site  to 
about  30  people,  then  conducted  assessment  team  training.  They  allowed  three  weeks  for  the 
assessment,  plus  another  week  for  the  training.  The  goal  was  to  evaluate  the  assessment 
method  and  not  to  focus  on  capability  levels  or  outcomes.  They  are  looking  at  22  process  ar¬ 
eas  (PAs).  They  are  performing  assessments  with  internal  people  and  providing  the  data  to 
the  SEI.  The  SEI  will  take  the  data  and  do  analysis  for  comparison  with  other  pilots.  A  focus 
of  the  pilot  is  trying  to  reduce  the  time  on  site  but  maintain  the  rigor  of  the  evaluation.  Also, 
they  will  have  SEI  observers  who  will  prepare  reports  during  assessment.  They  will  capture 
questions  about  the  model  as  well  as  the  assessment  method  (SCAMPI). 

5.4.6  CMMI  Representation  -  Staged  or  Continuous? 

There  was  a  brief  discussion  of  some  of  the  perceived  differences  between  the  staged  repre¬ 
sentation  and  the  continuous  representation  of  the  model. 

In  the  staged  representation 

•  the  concepts  can  be  communicated  clearly  with  senior  management 

•  there  is  an  element  of  simplicity  to  the  model  structure 

•  all  institutionalization  understanding  is  contained  to  process  areas 

•  if  an  organization  is  risk  averse  and  does  not  have  a  process  culture,  the  additional  elabo¬ 
ration  may  help  them 

•  the  structure  supports  top-down  process  improvement 

In  the  continuous  representation 

•  material  is  parceled  into  arbitrary  levels 

•  an  organization  can  pick  and  choose  an  area  and  focus  on  the  particulars  of  that  area 

•  the  21  processes  areas  in  Level  3  may  be  overwhelming  to  new  organizations 

•  an  organization  can  assess  progress  in  specific  process  areas  that  are  chosen  for  their 
business  value 

•  an  initial  assessment  may  provide  more  granularity  in  results  to  help  in  decision-making 
afterward 

There  was  a  discussion  of  some  of  the  organizational  and  environmental  factors  that  may  in¬ 
fluence  use  of  staged  vs.  continuous  representation. 
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The  staged  representation  works 


•  best  in  an  organization  with  strong  functional  orientation 

•  in  environments  that  typically  do  not  use  Integrated  Product  Teams,  and/or  software  is 
not  team-based 

•  in  organizations  where  the  management  is  far  removed  from  engineering  (i.e.,  a  closed 
organization  that  requires  push  to  management) 

•  for  organizations  that  cannot  use  data  well,  since  too  much  data  is  reported  back  from  the 
continuous  results 

•  for  very  large,  differentiated  organizations,  since  the  staged  results  are  more  easily  shared 
with  senior  management 

The  continuous  representation  works 

•  for  organizations  who  may  not  have  a  real  engineering  process  established/defined/ 
documented,  since  they  can  begin  in  designing  a  life  cycle 

•  for  organizations  who  have  sophisticated  engineering  and  products,  since  they  may  find 
the  granularity  and  incremental  change  beneficial 

•  for  organizations  that  are  team-based,  IPT-based,  and/or  management  is  very  close  to  en¬ 
gineering 

•  for  examination  of  a  very  focused  area,  with  few  levels  of  differences 

Organizations  who  come  to  CMMI  and  have  never  done  Software  CMM  will  approach  the 
model  representation  selection  process  differently.  Some  organizations  may  not  see  benefits 
of  the  staged  representation,  which  software  folks  take  for  granted.  Some  organizations  and 
customers  need  the  constraints  of  the  staged  representation;  while  others  find  they  cannot 
stand  staged. 

Organizations  with  experience  using  the  Systems  Engineering  CMM  and  continuous  assess¬ 
ments  with  a  Software  CMM  experience-base  react  differently  than  organizations  where 
software  and  systems  were  more  separate  and  using  different  models. 

When  doing  CMMI  and  communicating  adoption  principles  to  the  higher  executive  level, 
this  level  of  management  may  or  may  not  have  engineering  background,  model  knowledge, 
etc.  to  fully  appreciate  the  concepts  of  the  model  differences.  Executives  do  not  want  to  have 
to  make  decisions  about  subtlety.  If  an  organization  chooses  to  adopt  CMMI,  they  have  to 
figure  out  how  to  clearly  communicate  in  concepts  that  can  be  understood  by  senior  man¬ 
agement  (i.e.,  concepts  that  are  based  on  business,  not  models.).  In  an  organization  where 
Software  CMM  has  been  used,  senior  management  will  still  probably  be  conditioned  to  ask 
about  maturity  level  ratings. 


136 


CMU/SEI-2001-SR-014 


Even  though  there  are  multiple  representations,  it  is  not  necessary  that  an  organization  stay 
with  just  one  representation  or  methodology. 

Both  the  continuous  and  staged  representations  of  the  model  can  help  organizations  get  to 
maturity  level  5;  however,  the  model  does  not  help  organizations  go  beyond  Level  5.  The 
focus  beyond  Level  5  is  uncharted  territory  for  the  model.  This  may  require  going  back  to 
TQM  roots  and  looking  at  organization  goals. 

An  organization  may  choose  to  focus  on  improvement  of  observable  behaviors  by  applying 
the  generic  practices  from  CMMI.  They  can  be  used  to  communicate  and  work  with  im¬ 
provement  in  organizations  with  a  history  of  TQM.  TQM  was  not  based  on  clearly  observ¬ 
able  behavior;  however,  CMMs  contain  only  observable  behavior. 

The  CMMI  model  needs  to  be  used  and  understood.  Selection  of  the  representation,  or  de¬ 
termining  when  to  do  what  in  terms  of  process  improvement  implementation,  is  coupled  to 
both  culture  and  perspectives  of  the  organization  and  stakeholders.  The  model  helps  because 
an  organization  can  make  choices  even  within  the  model  for  improvement  priorities. 

5.4.7  Commercial  or  Other  Software-Only  Organizations 

How  will  CMMI  apply  to  commercial  or  other  software-only  organizations?  What  cautions 
and  opportunities  does  CMMI  provide  the  commercial  software-only  organizations?  One  or¬ 
ganization  has  been  doing  improvements  based  on  Software  CMM  principles  across  the  en¬ 
tire  company.  They  have  been  looking  at  CMMI  generic  practices  and  common  process  areas 
for  the  whole  company  and  use  the  engineering  Process  Areas  to  improve  where  applicable. 
They  develop  software  only,  but  they  still  have  problems  with  product  lines  and  problems 
with  requirements.  The  differences  between  software-only  companies  and  those  working  with 
large  systems  is  one  of  scale  rather  than  of  engineering  practices.  They  haven’t  seen  problems 
with  interpretation  of  practices — ^they  have  handled  interpretations  of  terminology  and  scal¬ 
ing  down  practices  to  work  in  a  small  company.  For  example,  they  use  a  general  Review 
Board  for  requirements  control,  configuration  control  board  (CCB),  and  process  reviews. 
They  relied  upon  a  former  SEI  staff  member  to  help  with  interpretation  during  the  first  year 
of  transition  from  Software  CMM.  After  that,  they  did  their  own  thing.  CMMI  makes  explicit 
what  they  were  doing  already  in  using  the  Software  CMM  principles  across  the  organization 
(i.e.,  the  generic  practices). 

What  part  of  CMMI  might  software-only  organizations  find  irrelevant?  Very  few  elements 
are  believed  to  be  irrelevant  to  most  organizations,  except  for  acquisition.  All  of  the  engineer¬ 
ing  practices  may  be  applied  to  software  only  organizations.  All  organizations  interpret  mod¬ 
els  to  satisfy  business  goals  and  objectives.  Differences  in  interpretation  come  from  differ¬ 
ently  sized  organizations,  or  those  with  different  outputs  or  products.  What  process  areas  are 
more  important  in  a  particular  company?  None  of  the  practices  are  unimportant — some  may 
be  more  important  or  implemented  differently,  depending  on  the  context. 
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Concern;  The  maturity  levels  have  gotten  very  large  (large  number  of  process  areas).  Is  it 
possible  to  extract  the  process  essentials  at  different  levels? 

Organizations  have  to  tailor  the  model  to  their  specific  context.  Determine  what  you  want  to 
accept  and  what  you  don’t  need.  This  is  the  key  to  making  the  CMMI  work  in  different 
organizations.  Is  it  easy  to  tailor  this  model?  If  you  don’t  want  to  use  part  of  the  model,  you 
should  document  your  reasons  so  you  can  explain  this  to  an  evaluator  or  assessor.  With  a 
large  project  and  teams,  you  can  take  slavish  obedience  to  CMMI.  Smaller  organizations  may 
need  expert  knowledge  in  tailoring  the  model.  CMMI  is  larger  than  Software  CMM,  but  it 
may  have  lost  some  of  the  essentials.  There  is  a  dichotomy  between  being  lean  and  providing 
information  that  helps  users  and  assessors.  Everything  in  CMMI  is  right  in  line  with  what  are 
called  “lean  practices,”  but  it  is  7(X)+  pages. 

In  1997,  high  maturity  organizations  were  concerned  they  were  losing  senior  management 
sponsorship  because  they  had  made  it  to  Level  5.  Still  true? 

These  days,  senior  management  sees  Six  Sigma,  Lean,  and  CMM  as  different  initiatives,  al¬ 
though  they  are  all  basically  the  same.  Six  Sigma  became  an  initiative  in  TQM  and  was  rec¬ 
ognized  to  be  of  value  beyond  software  and  system  engineering.  It  became  the  focus  of  all 
workforce  practices.  By  defining  defects  in  other  processes,  e.g.,  marketing.  Six  Sigma  be¬ 
came  the  umbrella,  and  for  software  and  system  engineering,  another  tool  for  SPC.  At  one 
organization,  all  senior  management  are  green  belts  in  Six  Sigma.  They  set  quantitative  goals 
for  their  areas.  Executives  that  came  from  different  backgrounds  and  from  different  compa¬ 
nies  are  now  all  working  together  under  this  umbrella. 

If  we  are  saying  that  the  basic  Software  CMM  improvement  process  can  be  tailored  for  any 
environment,  why  is  the  CMMI  model  different?  Why  is  it  bigger?  Case-specific  tailoring 
sometimes  leaves  out  specific  practices.  The  assessment  time  is  longer.  It  isn’t  clear  that  a 
given  organization  needs  to  adopt  all  of  the  practices  in  CMMI  and  whether  that  would  im¬ 
prove  the  bottom  line. 

What  practices  in  the  CMMI  are  not  applicable?  The  consensus  is  that  software  organizations 
will  apply  all  of  the  practices  in  CMMI.  How  is  it  too  heavy?  Implementation  of  CMMI 
should  be  focused  on  continuous  improvement  not  on  assessments.  CMMI  is  a  process  model 
not  Just  a  set  of  best  practices  to  evaluate  the  maturity  of  an  organization. 

Do  the  engineering  PAs  add  value  to  software-only  organizations? 

The  Risk  Management  PA  will  strengthen  weak  areas  that  haven’t  been  able  to  communicate 
to  senior  management.  The  continuous  model  with  its  focus  on  continuous  improvement  is 
opening  up  areas  for  process  improvement.  At  least  one  organization  is  using  the  CMMI  as  a 
checklist  for  finding  improvement  opportunities  in  their  current  engineering  processes. 
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Another  organization  adopted  the  Software  CMM  and  Systems  Engineering  CMM  with  one 
single  process  group.  They  had  been  continually  comparing  the  two  CMMs,  wanting  to  em¬ 
phasize  the  similarities,  and  CMMI  helps  with  this.  Now  everyone  is  working  toward  one 
model. 

CMMI  seems  to  be  providing  a  logical  extension  to  what  many  organizations  already  had  for 
software.  In  many  ways,  it  can  be  considered  a  kind  of  a  super-set  of  the  Software  CMM.  If 
so,  why  are  they  separate  programs  (Software  CMM  and  CMMI),  without  a  clear  progression 
from  one  to  the  other?  Why  does  Software  CMM  have  to  be  “sunsetted?”  Why  isn’t  there  a 
logical  progression  from  software  CMM  to  CMMI — training,  assessment,  everything?  We 
need  to  ask  the  SEI  this  question. 

5.4.8  Alternative  Practices 

Do  Level  5  organizations  develop  very  different  alternative  practices?  Are  there  differences 
based  on  organizational  structures  (e.g.,  hierarchical  vs.  flatter)?  This  issue  was  raised  in  the 
working  group;  however,  was  not  discussed  during  the  working  group  session. 

5.4.9  Recommendations  for  High  Maturity  Organizations 

The  group  discussion  expanded  to  cover  wider  maturity  with  respect  to  CMMI  adoption 
rather  than  just  higher  maturity.  Much  of  the  discussion  centered  on  strategic  issues  and  busi¬ 
ness  decisions  of  model  selection.  It  was  noted  that  CMMI  offers  the  wider  maturity  option 
and  a  broader  opportunity  for  integration  of  disciplines. 

As  a  result  of  the  working  group’s  discussions,  the  following  recommendations  for  organiza¬ 
tions  seeking  to  achieve  high  maturity  were  formulated: 

Recommendation  HM-1 — High  maturity  software  organizations  have  some  valuable  lessons 
learned  that  other  organizations  can  use  in  advancing  through  the  maturity  levels  and  as  other 
organizations  mature  through  CMMI.  Implement  measurements  early,  set  up  an  engineering 
process  group,  peer  reviews  and  other  forms  of  verification,  process  improvement  adoption 
lessons  learned. 

Recommendation  HM-2 — ^There  are  not  huge  differences  between  CMMI  and  Software 
CMM,  so  that  is  comforting.  Follow  TQM  principles  during  strategic  planning;  identify  mar¬ 
keting  areas  and  operational  direction.  If  you  have  an  initiative  that  crosses  the  organization 
(i.e.,  establishes  an  “umbrella”),  it  becomes  easier  to  deploy  CMMI  due  to  a  common  frame¬ 
work. 

Recommendation  HM-3 — Industry  has  to  examine  territory  beyond  maturity  level. 
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5.4.9.1  Observations 


In  addition  to  the  observations  above,  the  working  group  identified  a  number  of  other  obser¬ 
vations  relevant  to  high  maturity  organizations.  These  were: 

•  Level  of  impact  and  effort  in  a  high  maturity  organization  should  be  minimal  due  to  natu¬ 
ral  extension  from  Software  CMM  to  CMMI. 

•  Selection  of  the  Staged  versus  Continuous  implementation  may  depend  on  some  cultural, 
environmental,  and  management  factors. 

•  CMMI  can  be  especially  beneficial  to  organizations  with  less  mature  Systems  Engineer¬ 
ing  groups. 

•  CMMI  provides  commonality  in  process  improvement  across  Software  and  Systems  en¬ 
gineering  disciplines. 

•  CMMI  assessments,  formal  and/or  less  formal,  can  be  used  to  assess  the  feasibility  of 
application  of  the  CMMI  practices  and  assessment  method  for  an  organization. 

•  CMMI  generic  practices  can  be  successfully  applied  to  non-engineering  business  areas  to 
support  process  improvement. 

•  Basically  CMMI  has  broadened  the  base.  Implementation  has  more  to  do  with  size  of  the 
organization  than  with  disciplines  in  the  organization. 

It  was  also  noted  that  the  cost  for  process  assessments  is  very  high,  and  that  the  SEI  needs  to 
provide  a  less  expensive  and  less  time-consuming  assessment  method  for  CMMI.  The  three 
classes  of  planned  CMMI  assessments  were  briefly  discussed.  Class  A  assessments  reflect  the 
rigorous  process  used  in  order  to  achieve  ratings  and  proclaim  results  to  the  world.  The  Class 
B  and  C  assessments  are  designed  to  be  more  lightweight  methods  that  cost  less  but  are  a 
quick  check  of  where  an  organization  stands  against  the  model. 

5.4.1 0  Recommendations  for  the  SEI 

As  a  result  of  the  working  group’s  discussions,  the  following  recommendations  for  the  SEI 
were  formulated: 

Recommendation  SEI-1 — Why  is  CMMI  not  considered  Software  CMM  Version  3.0?  Why 
is  there  not  a  logical  progression  from  Software  CMM  to  CMMI  (in  models,  training,  and 
assessment  methods)?  In  lieu  of  such  a  progression,  organizations  will  have  a  more  complex 
transition. 

Recommendation  SEI-2 — Develop  a  CMMI  time-bound  release  plan  for  industry  involving 
all  aspects  of  the  organizations  (e.g.,  marketing.  Human  Resources,  etc.).  Take  an  enterprise¬ 
wide  assessment  approach,  e.g.,  Malcolm  Baldrige. 

Recommendation  SEI-3 — High  maturity  organizations  have  learned  how  to  quickly  and 
intelligently  implement  continuous  process  improvement.  Capture  those  lessons  learned  for 
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the  sake  of  others  and  provide  industry  a  road  map  to  get  through  the  model.  That  informa¬ 
tion  could  be  used  to  fine-tune  the  model  (e.g.,  case  studies). 

Recommendation  SEI-4 — Software  CMM  has  established  itself  as  an  international  de  facto 
standard.  It  is  not  desirable  to  risk  that  investment  by  a  badly  managed  transition  to  CMMI, 
such  that  the  user  community  loses  faith  in  CMMs.  The  defense/aerospace  industry  commu¬ 
nity  alone  cannot  keep  CMMI  alive  and  surviving;  it  has  to  be  accepted  around  the  world. 
Establish  industry-wide  support  and  buy-in,  including  involvement  from  the  commercial  sec¬ 
tor.  CMMI  has  been  focused  on  too  narrow  of  a  world  (initially).  Ensure  that  software-only 
organizations  can  see  that  the  model  works  for  them  too  and  that  there  are  clear  guidelines  of 
the  model  for  application  to  software-only  organizations. 

5.5  Working  Group  2.1 :  Statistical  Techniques 

The  participants  in  this  working  group  included  Joan  Romine,  Gerry  Ourada,  Jim  Vanfleet, 
Bill  Curtis,  Phil  Sperling,  Ashok  Sontakke,  Bruce  Boyd,  Anita  Carleton,  Dennis  Goldenson, 
and  Mark  Paulk. 

The  working  group  leader  was  Bill  Curtis.  The  facilitator/scribe  was  Dave  Zubrow.  The  re¬ 
corder  was  Will  Hayes. 

5.5.1  Hypotheses  and  Observations 

The  questions  that  inspired  the  creation  of  this  working  group  included: 

•  Do  high  maturity  organizations  use  control  charts?  XmR  charts?  U-Charts?  Othpr  charts? 
On  what  data?  On  what  processes? 

•  Do  high  maturity  organizations  use  regression  analysis?  On  what  data?  For  what  pur¬ 
pose? 

•  Do  high  maturity  organizations  use  tests  of  hypotheses?  On  what  data?  For  what  pur¬ 
pose? 

•  Do  high  maturity  organizations  use  multivariate  analysis?  On  what  data?  For  what  pur¬ 
pose? 

•  What  other  statistical  techniques  do  high  maturity  organizations  use?  On  what  data?  For 
what  purpose? 

•  What  statistical  techniques  (if  any)  must  an  organization  use  to  be  validly  considered 
high  maturity? 

•  What  is  the  business  value  of  statistical  techniques  that  has  been  observed?  Is  it  worth¬ 
while? 

During  initiation  of  the  working  group,  the  following  questions  or  observations  were  posed 
by  the  working  group  members: 
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•  Managers  generally  do  not  have  an  understanding  of  statistical  methods.  Training  in  basic 
statistical  concepts  and  methods  is  needed  before  introducing  Level  4  practices. 

•  Levels  4  and  5  and  Six  Sigma  need  to  be  integrated.  Six  Sigma  is  a  set  of  analysis  tools 
used  to  facilitate  process  improvement.  The  similarities  between  Levels  4  and  5  and  Six 
Sigma  are  too  great  to  treat  Six  Sigma  as  a  “new  approach.’'  The  software  community 
needs  to  better  understand  Six  Sigma  to  leam  how  it  can  best  be  used  to  direct  process 
improvement. 

•  The  working  group  was  in  complete  agreement  that  metrics  and  process  improvement 
activities  must  be  tied  to  business  results.  The  Goal-Question-Metric  (GQM)  approach  to 
defining  metrics  ensures  this  link.  Practitioners  need  to  keep  in  mind  that  the  success  of  a 
measurement  program  is  determined  by  the  successful  achievement  of  business  goals. 

•  Most  examples  demonstrating  the  use  of  statistical  techniques  involve  inspection  or  de¬ 
fect  data.  Statistical  process  control  has  primarily  been  focused  on  analysis  of  peer  re¬ 
view  data  (the  members  of  the  working  group  reported  use  of  u-charts  or  z-charts  to  ana¬ 
lyze  this  data).  The  software  community  needs  to  explore  how  other  types  of  data  (e.g., 
cost,  schedule,  reliability)  can  be  analyzed  to  improve  processes. 

The  issues  that  the  working  group  decided  to  discuss  in  more  detail  were 

•  Level  3  measures  often  not  adequate  to  support  Level  4 

•  What  is  required  for  Level  4? 

•  Relevant  statistics 

•  Use  of  organizational  capability  baselines 


5.5.2  Level  3  Measures  Often  Not  Adequate  to  Support  Level  4 

The  measures  used  by  Level  3  companies  generally  stay  at  the  project  phase  level  and  do  not 
provide  the  granularity  needed  at  Level  4.  The  Software  CMM  needs  to  provide  better  guid¬ 
ance  on  the  measures  to  have  in  place  at  Level  3  in  preparation  for  Level  4.  The  focus  for 
these  measures  needs  to  go  beyond  just  cost  and  schedule;  it  is  also  important  to  measure 
quality  at  Levels  2/3.  Measures  that  provide  valuable  insight  and  control  should  be  used  by 
all  organizations,  regardless  of  maturity. 

5.5.3  What  Is  Required  for  Level  4? 

Some  courses  and  consultants  have  stated  that 

•  Level  4  requires  control  charts 

•  every  process  must  be  under  statistical  control 

The  software  community  needs  to  understand  that  there  are  many  statistics  relevant  to  proc¬ 
ess  improvement.  Statistical  process  control  is  one  statistical  technique  that  has  its  purpose, 
but  it  should  be  treated  as  one  tool  in  a  “process  improvement  toolkit.”  Control  charts  done 
badly  (e.g.,  when  sources  of  variation  are  large  and  vary  widely  across  process  events)  is 
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worse  than  not  doing  SPC  at  all,  since  it  can  give  a  false  impression  of  process  stability  (large 
variation  with  little  predictive  value).  In  general,  classic  SPC  should  be  used  in  situations 
where  sources  of  variance  are  better  controlled.  It  should  be  used  because  it  is  the  correct 
analysis  technique,  not  just  to  achieve  Level  4. 

5.5.4  Relevant  Statistics 

Statistics  should  be  matched  to  the  purpose  for  the  analysis — prediction,  control,  or  under¬ 
standing  and  improvement.  High  maturity  organizations  have  a  limited  view  of  the  relevant 
statistics  to  achieve  these  purposes.  The  working  group  discussed  a  number  of  statistical  tech¬ 
niques  that  have  proven  to  be  useful  outside  of  software  development,  but  are  rarely  used  by 
Level  4  or  5  companies: 

•  Multivariate  methods  for  exploring  and  understanding  sources  of  variance 

•  Non-parametric  statistics  for  unusual  distributions 

•  Reliability  and  statistically  based  testing  for  determining  operational  performance  pro¬ 
files 

•  Bayesian  methods 

These  and  other  more  advanced  statistical  methods  should  be  explored  further  to  understand 
their  benefit  to  software  development. 

5.5.5  Use  of  Organizational  Capability  Baselines 

The  usefulness  of  organizational  capability  baselines  is  dependent  on  aggregating  data  at  the 
appropriate  level,  and  selecting  the  right  attributes  to  determine  which  projects  are  “similar.” 
However,  often  organization-wide  baselines  are  not  meaningful  to  projects.  Once  data  from 
many  projects  has  been  combined,  the  value  to  an  individual  project  (e.g.,  for  prediction  and 
control)  is  limited.  Therefore  process  performance  baselines  also  need  to  be  maintained  at  a 
lower  level  of  detail. 

The  members  of  the  working  group  reported  maintaining  process  capability  baselines  for  the 
following: 

•  productivity 

•  delivered  defects 

•  in-process  defects  (defect  profiles  by  phase) 

•  defects  per  LOC,  defects  per  hour  for  each  type  of  peer  review 

5.5.6  Recommendations  for  High  Maturity  Organizations 

As  a  result  of  the  working  group’s  discussions,  the  following  recommendations  for  organiza¬ 
tions  seeking  to  achieve  high  maturity  were  formulated: 
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•  Get  competent  statistical  guidance  with  experience  beyond  manufacturing. 

•  Let  the  data  and  objectives  determine  the  statistical  methods  used. 

•  Set  quantitative  objectives  tied  to  business  goals. 

•  Simplify  the  presentation  of  statistical  results. 

•  Use  data  to  gain  understanding  and  control  (Level  4)  and  guide  improvement  (Level  5). 

•  Learn  about  Six  Sigma  and  the  tools  it  offers  for  Level  4  and  5  activities. 

5.5.7  Recommendations  for  the  SEI 

As  a  result  of  the  working  group’s  discussions,  the  following  recommendations  for  the  SEI 
were  formulated: 

•  Improve  CMM/CMMI  to  focus  on  implementing  quantitative  management  rather  than 
focusing  too  narrowly  on  a  specific  method  of  implementation  (i.e.,  SPC). 

•  Perform  research  and  offer  training  on  quantitative  methods  in  addition  to  SPC. 

•  Continue  to  capture  and  disseminate  community  experience. 

•  Encourage  Level  4  and  5  organizations  to  publish  their  results  and  insights. 

•  Provide  better  guidance  and  training  for  Lead  Assessors. 

5.6  Working  Group  2.3  -  Change  Management  and 
People/Cultural  Issues  in  High  Maturity  Organiza¬ 
tions 

The  participants  in  this  working  group  included  Julie  Barnard  (United  Space  Alliance),  Lynn 
Carter  (SEI),  Eileen  Forrester  (SEI),  Bill  Hefley  (Q-Labs),  Christian  Hertneck  (Siemens), 

Bob  Hoekstra  (Philips  Software  Centre),  Judah  Mogilensky  (Process  Enhancement  Partners), 
Lynn  Penn  (Lockheed  Martin),  Lita  Schulte  (Boeing),  Somashekhar  R.H  (CG  Smith  Software 
Ltd.),  and  Gian  Wemyss  (SEI). 

The  working  group  leader  was  Bill  Hefley  (Q-Labs).  The  facilitator  was  Eileen  Forrester 
(SEI).  The  scribes  were  Diane  Gibson  (Carnegie  Mellon  University)  and  Lynn  Carter  (SEI). 
The  recorders  were  Gian  Wemyss  (SEI)  and  Christian  Hertneck  (Siemens). 

5.6.1  Hypotheses  and  Observations 

The  questions  that  inspired  the  creation  of  this  working  group  included: 

•  Does  your  high  maturity  organization  do  periodic  surveys  of  employee  satisfaction? 

•  Is  turnover  less  in  high  maturity  organizations?  Do  some  people  leave  the  organization  as 
a  result  of  disliking  the  process  discipline?  What  impact  has  that  had  on  organizational 
capability? 
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•  What  percentage  of  workers  participate  in  improvement? 

•  How  many  process  improvement  proposals  per  engineer  per  year  are  being  processed? 
Would  a  target  of  10  /  eng  /  year  be  reasonable? 

•  What  percentage  of  improvement  proposals  should  be  accepted?  Would  70-80%  be 
reasonable? 

•  What  latency  should  improvement  proposals  have  before  closure?  Would  five  working 
days  be  reasonable? 


Other  questions  that  were  raised  by  the  facilitator  from  the  SEI’s  Accelerating  Software 

Technology  Adoption  (ASIA)  initiative  included; 

•  In  your  organization,  is  the  motivation  for  adopting  technology  most  often  driven  by 
external  demand  (customer  needs)  or  internal  needs?  Both? 

•  What  are  the  characteristics  of  an  effective  “program”  of  technology  change  management 
(TCM)?  Are  good  TCM  programs  all  similar  or  do  they  vary?  Can  we  make  any  judg¬ 
ments  about  what  works  best  in  different  settings? 

•  Are  you  using  any  models  (besides  the  material  in  CMMs)  to  guide  your  implementation 
of  TCM?  What  has  proved  most  useful? 

•  Do  you  notice  any  differences  between  change  management  for  process  innovations  and 
for  all  other  types  of  technology?  What  metrics  do  you  find  most  useful  to  collect  on 
TCM? 

•  How  do  you  monitor  new  technologies? 

•  Do  your  TCM  practices  seem  to  work  well  for  both  disruptive  technologies  and  for  in¬ 
cremental  improvements? 


During  initiation  of  the  working  group,  the  working  group  members  posed  the  following 

questions  or  observations  relating  to  change  management: 

•  Cycle  time  for  process  improvement  activities 

•  Evidence  of  enterprise  wide  technology  /  process  change  management  approach 

•  Quantitative  evidence  of  process  improvement  proposal  participation  in  organization 

•  Constructs  of  automated  tool  for  managing  process  improvement  proposals 

•  Benchmark  data  on  processing  time  /  scope  /  size  of  process  improvement  proposals  from 
high  maturity  organizations 

•  Report  of  experiences  on  innovative  approaches  to  encourage  broader  participation  in 
submitting  process  improvement  proposals 

•  Understanding  of  how  TCM  (a  Software  CMM  maturity  level  5  key  process  area)  is 
operationalized  in  an  organization 

•  Capture  examples  of  TCM  practices 

•  Identify  candidates  for  TCM  studies 
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During  initiation  of  the  working  group,  the  working  group  members  posed  the  following 
questions  or  observations  relating  to  people  and  cultural  issues: 

•  Changing  in  the  face  of  change  (mergers  /  acquisition  /  reorganizations) 

•  Learn  what  barriers  to  change  (people  /  cultural  barriers  to  process  improvement)  have 
been  faced  by  organizations 

•  How  to  maintain  momentum  when  growing  big 

•  How  to  get  EVERYONE  involved  in  improvement 

•  Learn  ways  in  which  organizations  have  overcome  the  people  /  cultural  barriers  that  they 
have  faced 

•  Methods  used  to  increase  employee  satisfaction  with  process  improvement 

•  Retention  activities 

•  Employee  attrition  as  connected  to  high  maturity 

•  Are  yearly  surveys  enough? 

•  How  to  measure  management  maturity  and  their  impact  on  organization  maturity 

•  How  do  you  continually  measure  employee  satisfaction 

•  Methods  /  measures  used  for  employee  satisfaction 

•  Information  on  formal  adoption  of  employee  satisfaction  surveys 

•  Hear  pitfalls  /  issues  re:  people  and  change  in  high  maturity  organizations 

The  issues  that  the  working  group  decided  to  discuss  in  more  detail  addressed  both  change 
management  and  people  /  cultural  issues.  The  four  issues  in  these  two  categories  discussed  in 
more  detail  were: 

•  Change  Management 

-  Identification,  evaluation  and  monitoring  of  potential  changes 

-  What  is  an  effective  change  management  process?  (i.e.,  how  do  high  maturity  or¬ 
ganizations  make  change  happen?) 

•  People  and  Cultural  Issues 

-  People  /  cultural  /  value  issues  and  barriers 

-  Employee  attrition  /  retention  /  satisfaction 

5.6.2  Identification,  Evaluation  and  Monitoring  of  Potential 
Changes 

The  group’s  discussion  of  identifying,  evaluating,  and  monitoring  potential  changes  flowed 
into  a  broader  discussion  of  effective  change  management  processes,  of  which  identifying, 
evaluating,  and  monitoring  potential  changes  are  the  early  stages  of  a  robust,  effective  change 
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management  process.  The  topics  discussed  here  confirm  earlier  observations  regarding  high 
maturity  organizations  [Paulk  00],  which  include  the  following: 

•  Focus  of  improvement  activities  is  both  strategic  and  tactical,  while  clearly  understanding 
that  continual  improvement  depends  on  universal  participation. 

•  Apply  PDCA  cycles  and  Deming’s  “System  of  Profound  Knowledge.” 

•  Identify  and  select  new  technologies  and  process  innovations  based  on  organization’s 
process  improvement  goals,  business  case,  and  objective  criteria. 

•  Use  analytic  techniques  to  understand  the  impact  of  proposed  changes. 

•  Plan  and  manage  deployment  of  the  whole  change. 

•  Establish  measurably  better  and  better  processes  (and  products). 


5.6.3  Effective  Change  Management  Processes 

Organizations  represented  have  in  place  effective  change  management  processes;  each  tai¬ 
lored  to  their  unique  environments.  However,  in  discussing  the  question  “What  are  effective 
processes  (or  best  practices)?”,  it  was  evident  that  similar  process  architectures  for  the  change 
management  process  were  typically  implemented  in  these  organizations.  This  paragraph  pro¬ 
vides  a  summary  of  this  change  management  process,  which  is  shown  in  Figure  37. 


Identify  and  Select  Proposals 


Piloting 


Execution  of  Improvement  Plan 


Evaluate  &  Leverage  (Learning) 


Figure  37:  Change  Management  Process 

The  first  component  of  the  change  management  process  was  to  identify  and  select  improve¬ 
ment  proposals.  The  process  improvement  proposal  process  is  widely  communicated  across 
the  organization.  Without  communicating  this,  and  ensuring  that  individuals  know  that  there 
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are  ways  to  suggest  improvements,  organizations  do  not  always  ensure  that  they  will  over¬ 
come  a  key  barrier  to  effective  change  management — people  do  not  always  understand  that 
change  applies  to  them.  Some  organizations  reinforce  this  communication  and  solicitation  for 
improvement  proposals  by  employing  various  reward  mechanisms.  These  include  individuals 
receiving  a  lottery  entry  for  some  prize  as  a  result  of  submitting  suggestions  for  process  im¬ 
provement  or  other  similar  recognition  and  reward  mechanisms. 

Once  the  process  improvement  proposal  process  is  communicated,  it  must  utilize  appropriate 
sources  of  improvement  proposals  to  collect  improvement  proposals  from  across  the  organi¬ 
zation.  Numerous  sources  and  mechanisms  were  reported  to  be  in  use  by  high  maturity  or¬ 
ganizations,  as  shown  in  Table  5.  Once  the  improvement  proposals  are  received,  each  is  clas¬ 
sified  and  evaluated.  Where  appropriate,  pilot  activities  are  instituted  to  select  what  level  to 
institutionalize. 

Process  management  activities  deal  with  evaluating  the  proposed  changes  and  selecting  those 
to  implement.  Some  organizations  apply  process  modeling  and  simulation  to  model  the  cur¬ 
rent  process.  Data  on  current  process  performance  is  also  evaluated.  The  evaluation  examines 
performance  before  and  after  implementation  of  the  proposed  process  improvement.  Six 
Sigma  techniques  are  used  to  determine  what  part  of  the  process  to  “attack.”  Planning  for 
implementing  the  approved  improvement  proposals  is  begun,  and  plans  are  tracked  until  the 
process  owner  (or  relevant  individual)  closes  the  internal  improvement  plan.  Verification  of 
the  process  change  is  made  after  the  change  is  institutionalized  by  analyzing  the  before  and 
after  states  for  expected  values. 

The  second  component  of  the  change  management  process  addressed  piloting  or  trial  use  of 
the  proposed  improvements.  Improvement  proposals  are  classified,  and,  in  some  organiza¬ 
tions,  this  includes  a  determination  if  a  change  is  a  minor  or  a  major  change,  as  major 
changes  are  piloted.  Pilot  activities  are  planned,  when  appropriate.  For  identified  pilots, 
budgets  and  schedules  are  prepared  and  evaluation  activities  are  planned.  Appropriate  meas¬ 
ures  to  support  the  evaluation  of  the  proposed  improvement  are  developed.  Pilot  activities  are 
carried  out  to  determine  the  effectiveness  of  the  proposed  change,  how  best  to  implement  the 
proposed  change,  and  at  what  level  the  proposed  change  should  be  implemented. 


Table  5:  Sources  of  Improvement  Proposals 


Primary  Source  of  Improvement  Proposals 

Mechanisms  Used  to  Collect  Improvement  Pro¬ 
posals 

Internal  staff 

Quality  system  change  requests  (bottom-up) 
Improvement  proposals 

Interest  nets  (like  news  groups  supporting  discus¬ 
sion  and  measuring  interest) 

Expert  net  (mentors  or  coaches) 

Support  center  (SEPG) 

Request  for  tools 
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Primary  Source  of  Improvement  Proposals 

Mechanisms  Used  to  Collect  Improvement  Pro¬ 
posals 

Management 

Policy  deployment  from  high  level  goals  (top-down) 

Internal  and  external  benchmark  organizations 

Best  practices  quality  improvement  team,  perform¬ 
ing  internal  and  external  scans 

SEPG 

Process  audit  and  process  compliance  activities  as  a 
source  for  best  practices 

Technology  group 

Annual  technology  plan  -  technology  planning  as 
an  integral  part  of  business  planning  and  forecast¬ 
ing,  answering  the  questions,  “What  are  our  cus¬ 
tomers  expecting  us  to  offer  and  support?”  and 
“What  technology  is  needed  to  support  new  proc¬ 
esses?”  The  technology  plan  is  often  the  tactical 
implementation  of  the  strategic  plan. , 

The  third  component  of  the  change  management  process  is  to  implement  the  proposed  change 
according  to  the  approved  execution  or  implementation  plans.  Three  activities  were  identified 
as  essential  in  implementing  changes.  The  first  activity  is  training.  Training  issues  included 
not  only  considerations  for  providing  training  to  support  specific  changes  implemented,  but 
also  considerations  for  supporting  ongoing  process  training  needs.  Examples  of  these  other 
ongoing  training  mechanisms  used  included: 

•  building  a  discipline  of  reviewing  each  process  before  invoking  the  process,  ensuring  that 
only  the  most  recent  instances  were  invoked 

•  ensuring  that  each  individual  completed  an  annual  process  training  module,  which  is  a 
mandatory  process  training  event 

The  second  activity  is  executing  a  disciplined  process  for  managing  the  implementation,  in¬ 
cluding  release  management  considerations  for  controlling  releases  of  new  processes  or  tools 
into  the  workplace. 

The  third  activity  is  communicating  about  the  planned  change.  Communication  about 
changes  being  implemented  was  identified  as  being  quite  important.  Communication  topics 
covered  business  objectives  being  satisfied  or  supported  by  the  change,  notification  about  the 
changes  themselves,  and  information  about  training  to  support  the  change.  Specific  mecha¬ 
nisms  used  to  support  communication  about  processes  and  process  changes  include  using 
common  process  repositories,  distributing  automated  notification  of  process  changes,  and 
sending  weekly  change  email  for  changes  to  the  process  asset  library  (PAL). 

It  was  identified  that  communications  regarding  changes  is  quite  important,  as  it  can  be  a  key 
mechanism  for  overcoming  barriers  to  successful  implementation  of  changes.  Identified  bar¬ 
riers  that  communications  can  help  to  overcome  include  people  not  always  learning  of 
changes  and  people  not  always  understanding  that  the  change  applies  to  them. 

The  fourth  component  of  the  change  management  process  is  the  crucial  steps  dealing  with 
evaluating  and  leveraging  from  the  change,  or  learning  from  the  improvement.  This  is  ad- 
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dressed  in  high  maturity  organizations  both  at  the  level  of  evaluating  the  impact  of  individual 
improvements,  but  also  in  the  aggregate  to  evaluate  the  overall  process  trends.  Examples  of 
measures  used  in  these  evaluations  include  ROI  and  customer  satisfaction.  Other  measures 
are  addressed  below. 

In  parallel  with  all  the  other  change  management  activities,  the  organizations  maintain  an 
ongoing  set  of  process  management  activities.  These  activities  are  consistent  with  a  range  of 
process  management  endeavors  and  typically  involve  participants  from  the  organization,  the 
SEPG,  and  its  management  steering  committee.  Process  Management  activities  reported  in¬ 
clude: 

•  process  performance  analysis,  including  data  analysis  by  the  SEPG  of  data  from  individ¬ 
ual  projects 

•  process  coordination,  including  those  activities  associated  with 

-  the  process  owners  and  updating  and  refining  processes 

-  configuration  control  activities,  such  as  a  CCB  and  managing  the  periodic  releases  of 
updates  to  the  organization’s  set  of  standard  processes 

-  a  process  board  (e.g.,  a  management  steering  committee  or  enterprise  executive  steer¬ 
ing  committee) 

•  ongoing  communication  about  change  management  activities,  including  communication 
about  changes,  holding  process  improvement  forums  for  all  departments  (which  meet 
twice  a  month),  and  communicating  the  status  of  change  management  activities 


5.6.3.1  Essential  Change  Management  Functional  Roles 

A  number  of  roles  were  identified  as  being  essential  to  the  change  management  process. 

These  are  functional  roles,  each  having  a  specific  function  that  may  be  implemented  in  a  va¬ 
riety  of  different  ways,  depending  on  the  specific  organization.  These  roles  and  their  typical 

functions  are  as  follows: 

•  senior  manager/sponsor:  providing  leadership,  commitment,  and  support  for  process 
definition,  maintenance,  and  improvement 

•  strategic  planning:  setting  direction  for  the  organization  and  the  processes  it  must  deploy 

•  process  management:  manages  a  set  of  standard  processes;  responsible  for  process  per¬ 
formance  analysis,  process  coordination,  and  relevant  configuration  control  activities,  in¬ 
cluding  CCB 

•  scanning:  performs  benchmarks  and  monitors  external  sources  to  identify  best  practices 
or  potential  innovations  that  may  be  useful  to  the  organization 

•  process  owner:  defines  and  maintains  a  process.  At  the  organizational  level,  the  process 
owner  is  the  person  (or  team)  responsible  for  the  description  of  a  standard  process;  at  the 
project  level,  the  defined  process. 

•  process  consultant:  provides  subject  matter  expertise  and  consulting  on  specific  processes 
or  tools 

•  individual  initiator:  generates  or  initiates  improvement  proposals 
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Examples  of  these  roles  in  place  in  high  maturity  organizations  included  the  following: 

•  senior  manager/sponsor 

-  upper  management  leadership  and  support 

-  process  board  (enterprise  executive  steering  committee) 

•  strategic  planning 

-  technology  group  to  screen  technology  requests 

-  university  partners  forecast  technology  changes  and  needs 

-  corporate  research 

•  process  management 

-  SEPG  as  process  owner 

-  SEPG  analyzing  data  from  individual  projects 

-  change  control  board  (corporate  quality) 

-  process  control  board  (procedural  activities,  regionally  based) 

-  SEPG  as  CCB 

•  scanning 

-  TCM  committee  as  part  of  the  SEPG 

-  TCM  role  in  quality  management  systems 

•  process  owner 

-  SEPG  as  process  owner 

-  individual  people  as  process  owners  (autonomous) 

•  process  consultant 

-  subject  matter  experts 

-  SEPG  as  process  consultants 

•  individual  initiator 

-  process  improvement  forum,  all  departments  (meet  twice  a  month) 

-  task  force  as  improvement  owner 

-  inputted/initiator  of  process  improvement  proposals 

5.6.3.2  Change  Management  Measures 

Organizations  reported  using  a  wide  variety  of  change  management  measures.  These  meas¬ 
ures  addressed  outcomes  of  the  change  management  process,  the  status  of  the  change  man¬ 
agement  efforts,  evaluation  of  change  management  actions,  and  the  utilization  of  innovations 
or  improvements. 

Outcomes  measures  addressed  both  employees  and  customers,  as  well  as  the  impact  of 
changes  as  financial  measures  (e.g.,  ROI)  and  other  tangible  impacts.  Employee  satisfaction 
data  was  often  collected  at  multiple  levels,  including  a  corporate  level;  a  process  level  en- 
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compassing  those  who  use  a  process,  a  site  or  location,  and  a  specific  project.  Collected  met¬ 
rics  provided  insight  into  general  employee  satisfaction  as  well  as  satisfaction  with  the  proc¬ 
ess.  Asking  employees  for  this  latter  metric  was  sometimes  seen  as  a  demotivator,  however. 
Some  organizations  report  using  surveys  based  on  the  People  Capability  Maturity  Model  with 
both  manager  and  non-manager  populations  as  a  means  of  determining  outcomes  of  im¬ 
provement.  Financial  measures  were  addressed  in  the  represented  organizations  both  as  ROI 
of  proposed  changes  and  as  ROI  of  the  implemented  changes.  Other  tangible  impacts  of 
changes  were  determined  by  examining  the  impact  of  the  change  in  measures  of  productivity, 
defects  and  quality.  Customers  also  provided  outcomes  data  through  customer  satisfaction  or 
customer  intimacy  surveys,  as  well  as  interviews.  This  customer  data,  as  well  as  other  tangi¬ 
ble  data,  could  be  examined  using  trend  analysis  for  determining  the  impacts  of  the  changes 
implemented. 

Status  measures  were  used  by  high  maturity  organizations  to  track  the  volume  of  improve¬ 
ment  proposals  or  requests  for  assistance,  as  well  as  the  status  of  each.  In  one  organization 
that  used  a  formal  mentoring  structure  to  identify  improvement  needs,  one  measure  of  vol¬ 
ume  was  the  demand  for  experts  (i.e.,  mentors)  and  the  support  center.  Other  organizations 
track  the  number  of  process  improvement  proposals,  classified  and  reported  by  process  and 
their  severity  (i.e.,  major  or  minor).  Some  organizations  report  assigning  improvement  pro¬ 
posals  as  priority,  and  also  use  this  as  an  additional  category  in  tracking  improvement  pro¬ 
posals.  In  addition,  the  progress  of  improvement  proposals  is  tracked,  so  the  organizations 
know  the  status  of  each,  how  many  are  completed,  the  planned  and  actual  schedules  for  each, 
and  the  aging  (or  latency)  of  each  proposal  in  the  change  management  process.  A  drawback 
to  using  raw  measures  of  quantity  is  that  such  measures  are  not  always  normalized  per  person 
in  the  organization,  and  may  not  provide  a  meaningful  insight  into  the  true  volume  or  rate  of 
adoption  of  change  management  (i.e.,  continual  improvement)  practices  as  an  integral  part  of 
individual’s  work.  However,  those  organizations  using  PSP/TSP  have  specific  measures  of 
the  number  of  improvement  proposals  generated  by  each  trained  engineer. 

A  number  of  measures  were  used  to  support  evaluation  of  process  improvements.  These  in¬ 
cluded  standard  tool  evaluation  measures,  such  as  ROI,  the  usability  of  the  tool,  satisfaction 
of  current  tool  users,  and  data  indicating  the  volume  of  use.  Volume  of  use  could  address 
adoption  rates  organizationally  (i.e.,  number  of  projects  adopting  the  innovation)  as  well  as 
adoption  rates  within  each  project.  Other  measures  used  to  evaluate  potential  or  implemented 
improvements  included  simulations  of  processes.  Six  Sigma  techniques  were  also  used  to 
evaluate  and  narrow  down  options  to  the  most  effective  improvements.  Other  evaluation 
measures  were  based  on  pre-  and  post-  implementation  questionnaire  data.  In  one  organiza¬ 
tion,  these  questionnaires  were  based  on  a  standard  set  of  questions  from  a  repository  of  rele¬ 
vant  questions 

Utilization  measures  provide  another  means  to  gain  insight  into  the  processes  and  tools  used 
in  the  organization.  When  viewing  the  change  management  process  as  yet  another  process  in 
the  organization’s  set  of  standard  processes,  such  utilization  measures  also  provide  a  way  to 
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gather  data  about  the  change  management  process.  Utilization  measures  reported  in  use  ad¬ 
dressed  both  tool  utilization,  as  well  as  process  utilization.  A  variety  of  process  utilization 
measures  were  reported  in  use:  number  of  hits  on  a  Web  page  for  each  specific  process,  data 
available  from  process  non-compliance  reports,  and  data  available  from  a  process  compliance 
matrix.  This  last  approach  provided  insight  into  how  each  project  was  developing  its  project’s 
defined  process  based  on  the  organization’s  set  of  standard  processes.  Each  project  reported 
their  compliance  and  tailoring  of  the  organization’s  set  of  standard  processes  into  their  pro¬ 
ject’s  defined  process  via  a  compliance  matrix.  Centrally,  this  data  is  kept  in  a  relational  data¬ 
base  so  that  it  can  be  analyzed  quarterly  to  provide  insight  into  process  adherence  and  to 
identify  possible  improvements  to  the  organization’s  processes. 

5.6.3.3  Tool  Support  for  Change  Management 

A  number  of  tools  were  identified  across  high  maturity  organizations  as  being  in  place  to 
support  their  change  management  activities.  In  addition  to  metrics  databases  used  to  support 
the  measurements  above,  other  commonly  used  tools  included  integrated  repositories  of 
processes,  directories  of  subject-matter  experts  and  process  owners,  process  modeling  and 
simulation  tools  (e.g.,  BPWIN),  and  desktop  tools  used  to  submit  improvement  proposals. 
These  desktop  tools  took  a  number  of  forms,  depending  on  the  technology  in  use  in  the  or¬ 
ganization.  Various  forms  described  by  high  maturity  organizations  included  an  “Improve 
me”  button  on  a  screen  which  sent  an  e-mail  form  to  the  process  owner,  individual  process 
improvement  notes  or  proposals  captured  in  a  Lotus  Notes  repository,  or  an  enterprise-wide 
recommendations  template. 

5.6.4  People/CulturalA^alue  Issues  and  Barriers 

This  section  captures  the  working  group’s  discussions  that  covered  a  wide  range  of  peo¬ 
ple/cultural/value  issues  and  barriers.  Consistent  with  earlier  findings  [Hefley  99],  this  group 
identified  that  the  two  most  critical  people  issues  facing  their  organizations  were  retention 
and  turnover  and  enabling  their  staffs  to  deal  with  continual  change. 

5.6.4.1  Retention 

Retention  was  a  common  concern  to  high  maturity  organizations.  Retention  is  addressed  fur¬ 
ther  as  a  separate  issue  (see  “Employee  attrition/retention/satisfaction”  below). 

5.6.4.2  Change  Management  Barriers 

A  number  of  barriers  to  change  management  were  identified.  Many  of  these  are  the  standard 
concerns  one  thinks  of  when  addressing  change  in  human  organizations;  others  are  somewhat 
unique  to  high  maturity  organizations.  Standard  concerns  include  resistance  to  change  and 
individuals’  hesitation  to  change.  It  was  noted  that  it  is  important  to  explain  why  a  change  is 
being  made  and  the  context  of  the  proposed  change  to  overcome  this  inherent  resistance  to 
change. 
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Other  common  issues  noted  include  the  perception  of  process  as  overhead  or  that  there  has 
developed  a  perception  of  bureaucracy  involved  in  managing  change.  Some  organizations 
report  concerns  that  the  organization  is  undertaking  multiple  improvement  initiatives.  In 
some  cases  multiple  initiatives  are  seen  as  resulting  in  rapid-fire,  “flavor  of  the  month”  or 
“change  du  jour”  improvement  efforts  with  one  replacing  the  next,  while,  in  other  cases,  giv¬ 
ing  individuals  the  perception  of  a  high  volume  of  change  or  too  much  change  happening  in 
the  organization.  This  points  to  a  need  to  communicate  about  the  change  efforts,  as  the  group 
discussed  the  need  for  improvement  in  a  number  of  areas  (see  our  first  recommendation). 

Communication  is  clearly  important  is  overcoming  some  barriers.  People  need  to  understand 
that  they  have  the  time  to  change  while  doing  their  work.  People  do  not  always  learn  of 
changes,  and  they  do  not  always  understand  that  changes  apply  to  them. 

In  high  maturity  organizations,  there  is  a  risk  of  developing  a  certain  process  arrogance  or 
perception  that  “we  are  a  high  maturity  organization  and  we  have  our  act  together,  so  why  do 
we  need  to  do  more  process  improvement?”,  rather  than  developing  a  culture  of  continual 
improvement. 

5.6.4.3  Process  Motivation 

Organizations  considering  moving  to  higher  maturity  levels  need  to  be  clear  about  their  moti¬ 
vation  for  continual  process  improvement.  Some  organizations  strive  to  achieve  or  allegedly 
achieve  a  maturity  level  for  the  wrong  reasons,  such  as  having  an  organizational  badge  of 
honor.  In  such  organizations,  it  is  hard  to  sustain  continued  improvement,  as  people  aren’t 
bought  in  to  the  changes.  Some  organizations  pursue  higher  levels  of  organizational  maturity 
because  of  explicit  or  implicit  requirements  of  their  regional  culture  or  their  competitive 
situation. 

In  some  cases,  there  is  resistance  to  continued  improvement  beyond  maturity  level  3.  This  is 
often  manifested  as  resistance  because  attaining  higher  levels  of  maturity  is  “too  difficult”  or 
“doesn’t  apply  to  our  environment.”  Some  organizations  are  afraid  of  higher  customer  expec¬ 
tations  resulting  from  their  attaining  higher  maturity  levels.  This  is  somewhat  related  to  the 
understanding  of  “outsiders,”  such  as  customers,  about  what  to  expect  from  a  high  maturity 
organization.  Some  organizations  report  an  expectation  from  customers  that  high  maturity 
organizations  will  never  produce  a  defect  or  that  there  is  an  expectation  that  high  maturity 
means  they  must  be  perfect,  while  high  maturity  organizations  understand  that  even  high  ma¬ 
turity  processes  are  executed  by  humans,  and  are  capable  of  being  fallible  and  continually 
improved. 

Several  issues  were  addressed  as  both  motivators  and  demotivators  for  developing  and  sus¬ 
taining  a  process  focus  in  a  high  maturity  organization.  First,  as  a  motivator,  there  is  a  need 
for  higher  process  maturity.  In  some  cases,  this  is  driven  by  business  realities,  in  others;  it  is 
necessary  to  deal  with  turnover  (i.e.,  employee  departure)  rates  ranging  from  ten  to  fifty  per¬ 
cent  per  year.  When  experiencing  these  high  rates  of  turnover,  it  is  essential  to  have  defined 
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processes  in  place,  as  the  processes  are  what  will  carry  the  organization  forward.  In  many 
organizations,  the  people  themselves  want  high  maturity,  as  they  see  the  benefits  that  accrue 
as  a  result  of  attaining  higher  levels  of  maturity. 

As  demotivators,  some  organizations  report  that  they  continue  to  experience  resistance  to 
implementing  higher  levels  of  maturity  from  the  middle  of  their  ranks.  Mid-level  personnel 
are  sometimes  perceived  as  caught  with  a  double  burden  as  the  organization  matures,  not 
only  having  the  burden  of  managing  their  projects  and  producing  and  using  data  to  quantita¬ 
tively  manage  their  projects,  but  also  the  burden  of  producing  data  that  is  important  to  the 
organization  to  establish  organizational  capability  baselines  and  establish  process  perform¬ 
ance  baselines.  In  these  instances,  it  has  been  found  helpful  to  make  use  of  a  process  consult¬ 
ant  as  a  facilitator  to  support  learning  and  implementation  of  higher-maturity  quantitative 
management  practices  in  middle  management.  In  other  organizations,  it  has  been  found  less 
successful  to  move  this  function  to  a  quality  assurance  function,  giving  it  the  role  of  both  en¬ 
forcing  process  adherence  and  trying  to  cajole  reluctant  managers  into  using  the  processes. 

In  some  organizations,  there  was  reportedly  more  comfort  with  TCM  issues  than  with  PCM 
issues.  This  is  somewhat  natural,  given  that  software  engineering  has  traditionally  had  a  tech¬ 
nology-driven  nature,  and  organizations  may  feel  more  adept  at  coping  with  technology 
changes  than  with  process  changes.  Organizations  implementing  effective  change  manage¬ 
ment  processes  report  that  they  address  both  technology  and  process  changes  in  their  change 
management  processes. 

The  local  regional  culture  can  have  significant  impact  on  the  process  motivation  of  organiza¬ 
tions.  Areas,  such  as  Bangalore  with  multiple  high  maturity  organizations,  reinforce  attain¬ 
ment  of  higher-maturity,  while  the  culture  in  other  areas,  such  as  Silicon  Valley,  can  tend  to 
be  a  barrier  to  process  improvement. 

5.6.4.4  Culture 

Culture  issues  were  also  raised  involving  both  developing  a  culture  of  continual  change  and 
dealing  with  aspects  of  culture  as  a  component  of  a  merger,  acquisition,  or  reorganization. 
Three  issues  were  identified  as  important  in  developing  and  deploying  a  culture  of  continual 
change: 

1.  changing  in  the  face  of  change,  or  dealing  with  continual  need  for  change  and  multiple, 
continuing  change  efforts 

2.  acknowledging  change  and  those  who  do  it,  providing  communication,  recognition,  and 
reward  for  those  involved  in  all  aspects  of  change  management 

3.  intergroup  coordination,  which  is  addressed  in  the  next  paragraph 


Mergers,  acquisitions,  or  reorganizations  present  difficulties  (or  challenges)  for  high  maturity 
organizations.  They  present  issues  dealing  with  process  management,  such  as  how  much  of 
the  processes  should  be  retained  in  common  across  the  entire  new  organization,  what  should 
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be  the  standard  processes  for  the  new  organizations,  and  how  detailed  should  be  the  common 
processes  in  the  organization’s  set  of  standard  processes. 

Other  issues  in  mergers,  acquisitions,  or  reorganizations  arise  as  a  high  maturity  organization 
acquires  a  lower  maturity  organization  and  develops  a  long-term  process  management  road¬ 
map.  It  is  important  to  understand  the  consequences  of  the  merger,  acquisition,  or  reorganiza¬ 
tion  for  process  maturity,  as  well  as  any  potential  infrastructure  loss. 

There  are  also  issues  of  merging  these  different  cultures  and  reconciling  multiple  cultures. 
Some  organizations  have  expressed  the  need  for  a  tool  to  support  forms  of  “cultural  due  dili¬ 
gence”  to  understand  the  culture  within  the  acquired  organization.  High  maturity  organiza¬ 
tions  also  report  that  there  is  some  fear  of  losing  their  high  maturity  achievements  as  their 
culture  changes  through  the  merger,  and  the  need  to  establish  a  new  stable  level  of  capability 
from  the  newly  merged  processes. 

5.6.4.5  Boundary  Management 

Boundary  management  is  an  important  aspect  of  intergroup  coordination.  In  high  maturity 
organizations,  process  issues  affect,  and  sometimes  exacerbate,  boundary  management  issues. 
Like  cultural  mismatches,  some  of  these  boundary  management  issues  stem  from  differences 
in  process  maturity.  For  example,  issues  may  arise  on  integrated  product  teams  (IPTs)  for 
team  members  from  organizations  having  differing  levels  of  maturity.  High  maturity  team 
members  don’t  want  to  be  on  IPTs  with  low  maturity  team  members  for  fear  of  damage  to 
their  rating.  These  process  maturity  mismatches  also  occur  at  other  boundaries,  such  as  those 
that  the  organization  shares  with  its  customers,  suppliers,  and  teammates.  Some  of  these 
mismatches  are  truly  mismatches  between  levels  of  capability,  caused  by  differing  maturity 
levels,  while  others  are  process  mismatches,  caused  by  ineffective  or  non-existing  interfaces 
between  processes.  It  is  especially  important  when  these  issues  arise  that  a  shared  agreement 
on  roles  be  developed  and  put  into  place. 

5.6.4.6  Individual  Skills  for  Management,  Technical  and  Process  Man¬ 
agement 

Skill  issues  cut  across  all  aspects  of  developing  and  sustaining  a  high  maturity  culture.  The 
most  critical  aspects  of  these  issues  deal  with  developing  individuals  to  perform  effectively  in 
a  high  maturity  organization.  The  culture  (and  processes)  of  the  high  maturity  organization 
must  be  transferred  to  new  staff.  This  infusion  of  high  maturity  culture  to  new  staff  often  re¬ 
quires  that  new  staff  go  through  a  transition  to  shed  low  maturity  ways  of  doing  business,  and 
adopt  new,  higher  maturity  practices  and  skills.  There  can  be  a  lack  of  skills  or  a  gap  between 
people  and  roles  as  individuals  or  the  organization  adapt  to  a  higher-maturity  norm.  These 
issues  point  to  a  clear  need  to: 

•  understand  and  develop  the  competencies  needed  in  the  organization  as  it  moves  to  and 
sustains  higher  levels  of  maturity 
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•  gain  shared  agreements  about  roles  and  their  interactions,  as  well  as  knowledge  of  each 
other’s  roles  in  teams  in  high  maturity  organizations 

Bringing  new  employees  into  a  high  maturity  organization  presents  issues  in  addition  to  the 
skills-building  issue.  Some  new  employees  may  be  intimidated  by  the  high  maturity  nature  of 
their  new  organization,  and  fail  to  take  part  in  the  change  management  process.  These  indi¬ 
viduals  may  have  feelings  of  insecurity,  asking  themselves,  “Who  am  I  to  make  suggestions 
for  improvement?”  In  some  cases,  high  maturity  organizations  have  abandoned  hiring  outside 
management  staff  as  the  challenge  of  bringing  them  up  to  higher  maturity  performance  out¬ 
weighs  the  perceived  benefits.  This  shift  to  exclusively  hiring  managers  from  within  is  be¬ 
cause  the  cost  and  impact  of  bringing  in  outside  management  people  is  so  disruptive. 

5.6.4.7  Interpersonal  Skills 

Interpersonal  issues  that  can  arise  deal  with  maintaining  individuals’  perceptions  of  the  bal¬ 
ance  of  power,  and  gaps  between  people  and  roles.  Making  improvements  changes  relation¬ 
ships  between  people;  change  upsets  the  balances  of  power  /  influence.  Several  concerns  af¬ 
fect  people’s  perceptions  of  power.  These  include  a  reluctance  by  some  to  give  up  their  power 
over  people  and  move  to  management  by  fact,  concerns  by  individuals  about  losing  their  ex¬ 
pert  power  as  processes  become  institutionalized,  concerns  about  resisting  the  use  of  data  to 
manage  and  of  being  monitored.  Attaining  higher  levels  of  maturity  requires  developing  new 
skills  for  dealing  with  empowerment  for  both  individuals  in  the  organization  and  for  man¬ 
agement.  High  maturity  organizations  need  to  develop  ways  of  helping  people  who’ ve  been 
addicted  to  being  in  power  positions  to  accept  a  less  “powerful”  position  due  to  the  empow¬ 
erment  of  a  process  orientation  and  quantitative  management  by  facts. 

Gaps  between  people  and  roles  can  also  be  seen  by  the  resulting  symptoms,  which  appear  as 
interpersonal  issues  as  individuals  adapt  to  change,  new  roles,  new  processes,  and  new  ways 
of  high  maturity  management.  Appropriate  forms  of  training,  mentoring,  and  process  consul¬ 
tation  can  address  these  symptoms. 

5.6.5  Employee  Attrition  /  Retention  /  Satisfaction 

5.6.5.1  Retention  and  Attrition  issues 

Retention  is  a  very  common  issue  in  many  software  organizations.  This  is  also  true  of  high 
maturity  organizations.  Retention  must  be  managed.  One  organization  reported  that  manage¬ 
ment  sets  retention  goals  overall  and  for  regions.  Management  is  held  accountable  for  achiev¬ 
ing  these  goals  and  their  performance  in  doing  this  is  a  part  of  their  measurable  management 
objectives. 

Coupled  with  retention  is  the  opposing  side  of  the  coin:  attrition.  Attrition  will  always  be 
there,  so  organizations  must  have  ways  to  deal  with  it  and  overcome  it.  One  way  is  through 
attaining  higher  levels  of  process  maturity. 
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Some  attrition  in  high  maturity  organizations  is  perceived  to  be  useful.  In  these  cases,  those 
individuals  who  excel  at  being  “cowboys”  or  “heroes”  of  low-maturity  settings  or  who  are 
“in  love”  with  the  romance  of  Level  1  chaos  may  choose  to  leave.  These  individuals  do  not 
feel  comfortable  in  higher  maturity  settings,  and  their  leaving  creates  a  win-win  situation 
when  these  firefighters  leave  a  high  maturity  organization. 

In  other  cases,  people  who  leave  high  maturity  organizations  become  “boomerangs.” 
Through  their  initial  attrition,  they  create  long-term  retention  for  the  organization  as  they  re¬ 
turn  to  their  high  maturity  organizations  searching  for  the  high  maturity  environment  that 
they  had  left.  This  is  one  example  of  how  organizations  report  that  higher  maturity  helps  re¬ 
tain  both  people  and  knowledge. 

Software  professionals  have  been  shown  to  have  amongst  the  highest  growth  needs  among 
professions.  In  high  maturity  organizations,  “career  tigers”  shouldn’t  be  able  to  misuse  the 
system  as  much,  only  the  really  able  career  tigers  should  be  able  to  succeed  (measurably  by 
their  projects’  successes).  To  some  extent,  organizations  always  need  the  power  hungry  (or  at 
least  those  willing  to  deal  with  increased  responsibility).  This  should  be  supported  by  clear 
career  paths  for  these  career  tigers,  but,  at  the  same  time,  offering  possibilities  in  growth  for 
the  technically  oriented  people  who  may  not  want  to  pursue  a  management  career  path. 

Salary  was  discussed  as  a  related  issue.  It  was  commonly  perceived,  as  prior  studies  have 
shown,  that  salary  gets  an  employee  in  the  door  as  a  new  hire,  but  salary  alone  doesn’t  keep 
them  in  the  organization.  More  important  are  long-term  opportunities  for  employee  growth. 
Some  organizations  report  a  mismatch  of  expectations  of  college  recruits  for  growth  with 
reality. 

Some  organizations  reported  setting  regional  salaries,  based  in  part  on  regional  or  industry 
studies.  Others  report  salary  compression  issues,  salary  mismatches  across  comparable  posi¬ 
tions  after  mergers,  and  the  use  of  a  separate  critical  (or  “hot”)  skills  fund  in  addition  to 
flexible  merit  planning  for  salaries. 

5.6.5.2  Best  Practices  to  Support  Retention 

A  number  of  best  practices  were  identified  to  support  organization  retention  of  personnel. 
These  are  summarized  in  Table  6. 


Table  6:  Best  Retention  Practices  of  High  Maturity  Organizations 


Topic 

Best  Practice 

Communication 

Status  is  communicated  simultaneously  to  all  levels,  regardless  of  the 
management  chain 

Management  by  walking  around  (MBWA)  at  all  levels  of  the  organization 

Management  accessible  by  hosting  lunches,  conducting  round  tables, 
being  available  for  “skip  level”  interviews 
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Topic 

Best  Practice 

Performance  feedback  comes  to  individuals  through  360°  evaluations 

Career  Development 

Dual  career  ladders  available,  providing  support  for  growth  through 
management  and  technical  ladders 

Formal  mentoring  programs 

Job  rotation 

Teams 

Good  team  relations,  cohesion  and  loyalty 

Use  of  teaming 

Keeping  /  retaining  teams 

Team  formation  based  on  behavioral  characteristics  (e.g.,  Myers-Briggs 
Type  Indicator) 

Culture 

Sense  of  ownership 

Meeting-free  day  (i.e.,  a  business  day — not  Sunday  or  Saturday — that  is 
set  aside  as  a  meeting-free  day) 

5.6.5.3  Satisfaction/Retention/Attrition  Measures 

High  maturity  organizations  routinely  gather  and  use  measures  of  employee  satisfaction.  This 
is  implemented  in  a  number  of  different  ways  and  at  differing  periods.  Surveys  are  typically 
made  not  only  at  the  corporate  level,  but  also  at  lower  levels  in  the  organization.  These  lower 
levels  can  include  both  the  site  or  location  and  the  project  level.  These  local  surveys  allow  a 
focus  on  relevant  issues  that  may  not  affect  the  entire  organization.  Surveys  are  often  con¬ 
ducted  once  or  twice  annually,  while  some  organizations  use  a  monthly  or  quarterly  survey. 
Issues  addressed  in  these  surveys  include  employee  satisfaction,  satisfaction  within  group, 
technical  and  environmental  indicators,  satisfaction  with  management,  and  factors  relating  to 
stressors. 

Some  organizations  report  using  an  organizational  survey  based  on  the  People  CMM.  The 
People  CMM  gives  a  framework  for  assessing  organizational  workforce  practices.  In  some 
organizations,  questionnaires  based  on  the  People  CMM  are  perceived  as  resulting  in  more 
actionable  items  than  from  their  corporate  survey.  These  questionnaires  typically  provide 
coverage  of  most  of  the  People  CMM,  with  full  coverage  of  Level  2,  most  all  of  Level  3,  and 
major  components  of  Levels  4  and  5. 

Other  forms  of  evaluation  include  external  evaluations  by  customers,  and  a  “Dilbert  quo¬ 
tient,”  which  is  a  measure  of  the  number  of  Dilbert  cartoons  posted  in  the  organization. 

In  conducting  organizational  surveys,  high  maturity  organizations  understand  the  need  to  act 
on  what  you  find,  and  not  to  conduct  the  surveys,  if  the  organization  is  not  willing  to  act  on 
results.  These  surveys  provide  an  additional  means  of  identifying  improvement  needs  or  of 
identifying  potential  problems.  As  an  example  of  such  results.  Figure  38  shows  the  results  of 
an  internal  team  survey  as  a  “Happy  Team  Index.”  Data  were  collected  in  a  monthly  survey 
in  the  project,  collected  into  a  simple  MS-Excel  application.  Evaluation  of  the  data  was  done 
by  the  project’s  quality  assurance  staff. 
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Figure  38:  Results  of  Internal  Team  Survey 

In  this  example,  the  survey  is  based  on  questions  relating  to  a  technical  HTI,  addressing  is¬ 
sues  such  as  technical  communication,  environment,  and  the  organization;  and  a  personal 
HTI,  addressing  the  individual’s  personal  influence  of  project  (communication,  relation  to 
colleages  and  customers)  and  feelings.  Each  of  the  questions  in  these  categories  is  rated  from 
I-IO  (with  10  being  the  best). 

The  above  diagram  shows  an  example  of  a  troubled  project  in  a  lower  maturity  organization. 
It  depicts  a  situation  you  don’t  want  to  see  in  your  project:  Increasing  team  size  coupled  with 
decreasing  participation  in  the  monthly  surveys,  while  the  perceived  project  situation  was 
being  reported  by  respondents  as  actually  becoming  worse  each  month  (as  rated  by  personal 
HTI).  Even  though  the  technical  environment  (technical  HTI)  seemed  to  be  getting  better 
within  the  project,  the  real  problem  was  the  internal  communication  to  management  and  the 
customer.  This  project  ended  in  a  disaster  (with  millions  lost  and  a  lawsuit  pending). 

High  maturity  organizations  also  report  measuring  management  effectiveness.  In  these  set¬ 
tings,  this  feedback  is  part  of  manager’s  measurable  business  objectives.  Several  techniques 
are  used  for  collecting  this  feedback.  These  include  focused  surveys,  feedback  on  manage¬ 
ment  from  team  members  (every  6  months),  surveys  based  on  the  People  CMM,  and  360° 
evaluations. 

Another  example  of  a  template  used  to  collect  feedback  on  management  from  team  members 
is  shown  in  Figure  39. 
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This  is  part  of  the  evaluation  at  High  Maturity  Organization  (HMO)  to  make  sure  that 
are  we  meeting  the  expectation  of  the  internal  customer  (i.e.,  you).  Feel  free  to  voice 
your  opinions  so  that  we  can  improve  and  provide  better  support,  which  goes  a  long 
way  in  making  HMO  a  Mecca  for  software  engineers.  We  request  your  comments  for 
the  ratings  to  enable  us  to  further  work  on  all  aspects  of  Total  Internal  Customer  Satis¬ 
faction. 


The  rating  scale  for  the  group  ranges  from  1  to  5: 

1 -Unacceptable  2-Needs  Improvement  3-Average  4-Good  5-Excellent 

Process  and  Technology  Group  (SEPG)  (Process  support,  tools,  etc.) 

1  2  3  4  5 

O  O  O  O  O 

Comments: 


Human  Resources  Dept.  (Recruitment,  training,  grievances,  transport,  etc.) 
1  2  3  4  5 

O  O  O  O  O 

Comments: 


Finance  Dept.  (Salary,  travel,  reimbursement,  etc.) 
1  2  3  4  5 

O  O  O  O  O 

Comments: 


Administration  Dept.  (Facilities,  parking,  coffee/tea,  cafeteria,  etc.) 
1  2  3  4  5 

O  O  O  O  O 

Comments: 


System  Administration  Dept.  (PCs,  printers,  software,  services,  telephones,  etc.) 
1  2  3  4  5 

O  O  O  O  O 

Comments: 


Feedback  regarding  Project  Members 
1  2  3  4  5 

O  O  O  O  O 

Comments: 

Feedback  regarding  Supervisors  (Project  Coordinators,  Project  Leaders,  Project  Man¬ 
agers,  Vice-President  etc.,) 

1  2  3  4  5 

O  O  O  O  O 

Comments: 


Figure  39:  Management  Feedback  Template 


CMU/SEI-2001-SR-014 


161 


Employee  satisfaction  and  process  maturity  are  inextricably  linked.  In  high  maturity  organi¬ 
zations,  it  is  important  to  have  personalized  attention,  as  individuals  are  contributing  mem¬ 
bers  of  high-performing  teams,  and  not  just  robots  in  a  software  factory.  However,  there  may 
be  resistance  to  asking  these  questions  in  some  settings,  as  asking  about  employee  satisfac¬ 
tion  with  processes  can  get  a  very  negative  reaction  in  a  process-disciplined,  well-established, 
high  maturity  organization  that  has  numerous  functioning  mechanisms  to  surface  and  correct 
process  problems. 

5.6.6  Recommendations  for  High  Maturity  Organizations 

5.6.6.1  Recommendations 

As  a  result  of  the  working  group’s  discussions,  the  following  recommendations  for  organiza¬ 
tion  seeking  to  achieve  high  maturity  were  formulated: 

Recommendation  HM-1 — Work  force  management  processes  must  mature  commensurate 
with  other  disciplines;  the  workforce  expects  high  maturity  in  all  areas. 

Numerous  organizations  are  using  the  People  CMM  as  a  guide  in  maturing  their  workforce 
practices. 

Recommendation  HM-2 — Use  broad  range  of  sources  for  improvement  ideas. 

Examples  of  the  sources  used  by  organizations  are  found  in  Table  5. 

Recommendation  HM-3 — Explore  how  to  do  “cultural  due  diligence”  to  handle  the  pressure 
and  risk  of  acquisitions  and  mergers. 

Some  organizations  report  use  of  the  People  CMM  to  support  this  need  for  cultural  due  dili¬ 
gence.  Although  a  People  CMM-based  assessment  method  exists  [Hefley  98],  a  lightweight. 
Class  B  or  C  appraisal  technique  that  provides  a  common  organizational  measure  could  sup¬ 
port  this  need  to  provide  a  characterization  of  the  workforce  capability  of  organizations. 

Recommendation  HM-4 — Establish  resources  and  energy  for  process  improvement. 

An  organization’s  process  improvement  activities  should  actively  solicit  improvement  pro¬ 
posals,  encourage  participation  in  proposing  and  implementing  change,  and  reward  participa¬ 
tion  in  effective  change  management  activities. 

Recommendation  HM-5 — If  an  organization  asks  its  people  to  participate  in  change  activi¬ 
ties,  it  should  make  it  easy  by  providing  appropriate  support  (tool  support,  etc.)  and  make 
sure  it  provides  a  benefit  to  the  people. 
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5.6.6.2  Observations 

In  addition  to  the  recommendations  above,  the  working  group  identified  a  number  of  other 

observations  relevant  to  high  maturity  organizations.  These  were: 

•  Organizations  striving  to  become  high  maturity  organizations  need  to  develop  a  culture 
of  continuous  improvement  throughout  the  whole  organization. 

•  Participation  in  improvement  is  essential  across  the  entire  organization. 

•  The  functional  roles  identified  by  this  working  group  need  to  be  assigned  and  in  place  for 
effective  change  management.  While  this  report  has  provided  multiple  examples  of  how 
to  assign  these  roles  to  manage  change  effectively,  the  actual  roles  and  responsibilities  in 
the  organization  should  be  clearly  assigned  and  communicated. 

•  Effective  evaluation  of  improvements  requires  understanding  of  the  organization’s  capa¬ 
bility  baselines  as  well  as  the  impact  of  changes. 

•  High  maturity  organizations  need  to  be  prepared  to  manage  transition  when  working  with 
lower  maturity  organizations  and  members. 

•  Organizations  should  not  address  change  with  out  addressing  the  people  issues.  These 
issues  should  be  addressed  in  planning,  implementation,  and  evaluation  of  changes.  In 
evaluating  the  impact  of  changes  on  people,  consider  a  wide  variety  of  things  to  measure, 
such  as  employee  satisfaction,  technical  satisfaction,  stress  factors,  and  management  ef¬ 
fectiveness. 

•  If  an  organization  has  high  attrition,  it  needs  high  maturity  practices  to  survive. 


5.6.7  Recommendations  for  the  SEI 

As  a  result  of  the  working  group’s  discussions,  the  following  prioritized  recommendations  for 
the  SEI  were  formulated: 

Recommendation  SEI-1 — Consider  hosting  an  explicit,  ongoing,  invitation-only  forum  tar¬ 
geted  to  senior  management,  for  example,  “a  sponsor  workshop”  to  cover  roles  and  skills  for 
sponsors. 

Recommendation  SEI-2 — ^There  should  be  a  “What  Does  Management  Look  Like  in  High 
Maturity  Organizations?”  course. 

Recommendation  SEl-3 — A  group  from  this  workshop  should  work  with  the  SEI  to  capture 
best  practices  in  change  management  because  we  heard  considerable  consistency  in  what 
works,  and  this  common,  best  practice  could  be  captured  and  disseminated  throughout  the 
community. 

Several  participants  from  this  workshop  have  volunteered  to  participate  in  preparing  a  best 
practices  guide  for  change  management.  The  SEI  should  consider  publishing  this  guide. 
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Recommendation  SEI-4 — Define  and  support  reasonable  expectations  of  Level  4  and  5  or¬ 
ganizations — better  educate  customers  on  concepts  of  process  capability  vs.  process  perform¬ 
ance. 

Recommendation  SEI-5 — The  SEI  should  more  consistently  consider  global  issues.  For  ex¬ 
ample,  consider  holding  the  next  high  maturity  workshop  in  Bangalore,  India. 

Recommendation  SEI-6 — The  SEI  might  try  to  become  a  high  maturity  organization  or  pro¬ 
vide  more  working  examples  of  high  maturity  organizations. 

5.6.8  Conclusion 

Effective  change  management  processes  that  consider  people,  process,  and  technology  can 
reinforce  an  organization’s  evolving  and  optimizing  culture  of  continual  improvement.  Even 
these  change-management  processes  are  components  of  the  organization’s  set  of  standard 
processes  and  can  (and  should)  be  continually  improved.  Paying  attention  to  the  people  is¬ 
sues,  as  well  as  the  process  and  tools  issues,  is  essential,  as  even  Level  5  processes  are  exe¬ 
cuted  by  humans! 

5.7  Working  Group  2.4:  Process  Agility,  Internet 
Speed,  and  Process  Improvement 

The  participants  in  this  working  group  included  Vivek  Govilkar,  Subrata  Guha,  Raj  Shekhar, 
Anthony  D’Souza,  Albert  Soule,  Jitendra  Shreemali,  Muthuramalingam  Rajamanickam,  Jim 
McHale,  Suzie  Garcia,  Linda  Levine,  and  Gangi  Keeni. 

The  working  group  leader  was  Gangi  Keeni.  The  facilitator  /  scribe  was  Suzie  Garcia.  The 
recorder  was  Linda  Levine. 

5.7.1  Hypotheses  and  Observations 

The  original  questions  that  inspired  the  creation  of  this  working  group  included  questions  on: 

•  Internet  speed  and  process  improvement: 

-  Do  “lightweight”  or  “Internet-speed”  processes  such  as  XP  and  Scrum  substantively 
address  Software  CMM  practices? 

-  Can  an  organization  be  Level  2+  if  it  has  a  high  percentage  of  projects  following 
lightweight  processes? 

-  Are  there  specific  practices  in  “lightweight”  processes  that  would  be  considered  un¬ 
acceptable  in  a  mature  software  process? 

•  Process  agility 

-  Is  a  high  maturity  organization,  by  virtue  of  its  process  discipline,  more  conservative 
with  respect  to  process  change? 
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-  Can  a  high  maturity  organization,  with  all  its  infrastructure,  change  as  rapidly  as  the 
business  environment  may  demand? 

-  Should  high  maturity  organizations  focus  on  “lightweight”  processes? 

First,  the  group  covered  introductions  and  also  described  the  level  of  experience  each  had 
with  process  agility  and  Internet  speed  issues:  typical  project  and  project  size. 

5.7.1 .1  Early  discussion 

During  initiation  of  the  working  group,  the  following  questions  or  observations  were  posed 
by  the  working  group  members. 

The  group  did  not  have  much  experience  in  lightweight  methodologies  such  as  XP.  There 
were  discussions  about  what  is  a  lightweight  process;  does  lightweight  imply  specific  lack  of 
practices.  Process  Change  Management  implies  continuous  improvement,  which  may  result 
in  lightweight  processes  to  suit  the  business  requirement. 

The  group  had  lots  of  experience  with  tailoring  standard  processes  for  short-duration  projects 
(1-3  months)  and/or  creating  standard  processes  for  short  duration  projects.  The  definition  of 
“light”  was  expanded  to  include  these  homegrown  processes  as  well.  The  group  used  “agile” 
in  place  of  “light”  for  most  of  the  discussions. 

The  group  decided  to  concentrate  on  the  following  questions: 

1 .  How  can  Software  CMM  be  relevant  for  organizations  doing  only  short-duration  pro¬ 
jects?  Or  those  who  are  using  only  light  processes  such  as  XP? 

2.  How/do  high  maturity  practices  help/hinder  organizations  in  performing  short  duration 
projects? 

3.  How  do  you  leverage  experience  in  high  maturity  into  Internet  speed  projects? 

4.  Do  “lightweight”  or  “Internet-speed”  processes  such  as  XP  and  Scrum  substantively  ad¬ 
dress  Software  CMM  practices? 

5.  Are  there  specific  practices  in  “lightweight”  processes  that  would  be  considered  unac¬ 
ceptable  in  a  mature  software  process? 

6.  Should  high  maturity  organizations  focus  on  “lightweight”  processes? 

7.  Is  a  high  maturity  organization,  by  virtue  of  its  process  discipline,  more  conservative 
with  respect  to  process  change? 

8.  Can  a  high  maturity  organization,  with  all  its  infrastructure,  change  as  rapidly  as  the 
business  environment  may  demand? 

9.  Which  Software  CMM  practices  help/hinder  you  with  short  duration  projects? 

10.  What  are  the  major  differences  in  your  “light”  and  “normal”  processes? 

1 1 .  What  problems  have  you  found  with  regard  to  Software  CMM  adherence  or  business 
results  when  trying/using  your  “light  processes? 
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12.  How  is  your  ability  to  respond  to  rapidly  changing  business  demands  helped/hindered  by 
your  high  maturity  practices? 


The  issues  that  the  working  group  decided  to  discuss  in  more  detail  were: 

•  Which  Software  CMM  practices  help  you  with  short-duration  projects? 

•  What  are  the  major  differences  in  your  “light”  and  “normal”  processes? 

•  What  problems  have  you  found  with  regard  to  Software  CMM  adherence  or  business  re¬ 
sults  when  trying/using  your  “light  processes?” 

•  How  is  your  ability  to  respond  to  rapidly  changing  business  demands  helped/hindered  by 
your  high  maturity  practices? 

5.7.2  Software  CMM  Support  for  Short-Duration  Projects 

The  following  KPAs  have  a  key  role  in  successful  execution  of  short-duration  projects. 

•  ISM  (Integrated  Software  Management)  -  Tailoring  is  the  key  for  short-duration  projects 

•  SPE  (Software  Product  Engineering)  -  Tailoring  of  SPE  practices  based  on  data  to  ana¬ 
lyze  risk 

•  PCM  (Process  Change  management)  /  TCM  (Technology  Change  Management)  -  Pilot¬ 
ing  practices  to  reduce  the  risk  and  get  better  insight 

Though  these  KPAs  have  an  explicit  impact,  it  was  felt  that  the  experience  of  all  Software 
CMM  practices  help  in  effective  tailoring  and  understanding  of  the  risks  involved.  As  the 
maturity  increases,  the  granularity  in  measurements  generally  shift  to  enable  better  insight  in 
the  processes.  The  Software  CMM  focus  on  measurement  encourages  one  to  look  at  the  right 
granularity  for  short-duration  projects. 

Having  a  family  of  standard  processes,  instead  having  a  single  standard  process  was  consid¬ 
ered  to  be  an  advantage  in  this  context  (e.g.,  there  could  be  one  for  short-duration  projects). 

5.7.3  Agile  Processes  for  Short-Duration  Projects 

The  participants  were  of  the  view  that  these  processes  could  be  a  methodology  or  just  se¬ 
lected  processes  adjusted.  However  it  would  be  defined  well  enough  to  be  trained/performed 
by  new  people.  Light  is  not  just  leaving  things  out  or  skipping  something;  sometimes  the  ac¬ 
tivity  may  be  diluted,  which  starts  out  as  tailoring.  It  is  a  different  process,  but  it  may  be 
about  degree  of  formality.  Eventually,  an  agile  tailored  process  may  become  “standard.” 

Organizations  may  trade  off  their  current  optimal  set  of  processes  for  a  lighter  process  that  is 
less  rigorous  than  today’s  best  processes  in  order  to  reap  other  benefits,  such  as  time  to  mar¬ 
ket. 
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Today’s  “light”  process  (less  than  optimal)  may  become  tomorrow’s  optimal  process,  because 
of  the  learning.  Here,  the  risks  are  accepted  because  of  the  ability  to  quickly  respond  to  prob¬ 
lems. 

It  was  highlighted  that  in  context  of  PSP/TSP,  one  of  the  things  that  Watts  Humphrey  always 
emphasizes  is  that  the  fastest  way  of  doing  something  is  almost  always  the  right  way  of  doing 
something.  And  not  everything  worth  doing  is  worth  doing  well. 

Some  of  the  helpful  practices,  for  short-duration  projects,  discussed  were: 

•  Project  startup,  where  all  the  performers/stakeholders  and  support  groups  such  as  SEPQ 
meet  and  decide  on  the  approach  to  be  taken.  The  risks  are  discussed  and  the  parties 
come  to  a  common  agreement.  The  SEPG  would  be  involved  in  tracking  new  processes 
or  any  deviation  from  the  organization’s  standard  software  processes  and  would  initiate 
corrective  actions  as  required. 

•  online  review:  whereby  the  turnaround  of  review  processes  is  reduced  to  real  time 

•  automation  for  many  system  processes  (especially  measurement)  makes  it  transparent 

•  defect  prevention  is  never  tailored  out 

5.7.4  Problems  in  Software  CMM  Adherence  in  Short-Duration 
Projects 

The  discussions  here  mainly  concerned  fully  meeting  the  Level  4  KPA  requirements  (e.g.,  is 
the  process  measurement  driving  the  process  change?). 

•  What  tailoring  is:  cutting  things  out,  finding  shortcuts,  and  shrinking  things  down. 

•  What’s  a  lightweight  process? 

•  Maintenance  may  be  short  duration  but  it  is  quite  different  from  the  Internet  projects. 

The  above  recurring  issues  led  the  group  to  reiterate,  how  short  is  short?  When  is  a  project 
too  short  to  be  a  project?  And  the  group  agreed  on  the  following: 

•  Each  and  every  emergency  bug  fix  is  not  a  project  by  itself. 

•  Maintenance  project  enhancements  bundled  into  a  time  box  can  be  treated  as  a  project. 

•  A  four-day  Web  site  development  with  six  people  can  be  treated  as  a  project  by  itself. 

•  To  be  a  short-duration  project,  the  project  should  have  goals  and  delivery  criteria  similar 
to  larger  projects  that  can  be  planned  and  measured. 

•  There  may  be  different  granularity  of  measurements  but  it  is  still  required  to  be  able  to 
identify  control  points. 
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5.7.5  Ability  to  Respond  to  Rapidly  Changing  Business 
Demands 

While  discussing  the  business  pressure  for  speed  it  was  discussed  that 

•  in  some  environments,  lower  quality  might  be  tolerated  with  innovative  products  for  first 
to  market 

•  in  many  environments,  tolerance  for  low  quality  is  not  there 


These  are  different  categories  altogether  and  need  to  be  addressed  accordingly.  However,  all 
agreed  that  data  is  the  key  in  making  decisions  in  this  regard. 

•  Data  allows  one  to  see  trends  from  the  outside. 

•  Data  helps  create  realistic  expectations. 

•  Data  allows  more  experimentation  and  fast  feedback. 

5.7.6  Recommendations  for  High  Maturity  Organizations 

As  a  result  of  the  working  group’s  discussions,  the  following  recommendations  for  organiza¬ 
tions  seeking  to  achieve  high  maturity  were  formulated: 

•  Participate  in  SEI  research  in  this  area. 

•  Publish  and  share  learnings  on  agile  high  maturity  organizations. 

-  business  drivers 
“  measures 

-  best  practices 

-  case  studies,  experience  reports 

•  Contribute  to  a  common  database  (industry  wide).  Include 

“  information  on  techniques 

-  experiences 
“  experiments 

benchmarking  data 

•  Encourage  the  community  to  make  data  available  to  the  Software  Engineering  Informa¬ 
tion  Repository  (SEIR). 

•  Keep  an  open  mind,  evaluate  new  trends.  Propose  changes  as  relevant  to  the  SEI. 

•  Balance  rigor  with  agility  when  defining  project  processes. 

•  Gather  and  use  data  to  understand  risks  and  consequences  associated  with  agile  proc¬ 
esses. 
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5.7.7  Recommendations  for  the  SEI 

As  a  result  of  the  working  group’s  discussions,  the  following  recommendations  for  the  SEI 
were  formulated; 

•  Provide  a  better  definition  of  “light”  if  this  topic  is  intended  to  be  studied. 

•  Sponsor  a  study  of  high  maturity  organizations  that  are  doing  Internet  development  pro¬ 
jects. 

•  Sponsor  follow-on  work  to  existing  exploratory  study  on  Internet  speed. 

•  Sponsor  a  user  group  (high  maturity  organizations)  to  consider  Internet  speed  and  other 
topics. 

5.8  Working  Group  2.7  -  CMMI 

Because  of  the  interest  in  CMMI,  Working  Group  1.5  on  CMMI  was  duplicated  in  the  second 
set  of  working  group  sessions.  The  participants  were  Donna  Dunaway  (Facilitator),  David 
White  (Scribe),  Joseph  Morin,  Mel  Wahlberg,  Wendy  Irion  Talbot  (Scribe-flip  charts),  Mary 
Beth  Chrissis,  Roger  Bate,  John  Yu,  and  Asha  Goyal  (Leader).  Visitor:  Mark  Paulk. 

5.8.1  The  Model 

There  is  a  large  community  using  the  Software  CMM  model  and  a  transition  has  to  be  built  in 
to  any  new  model  or  new  version  of  the  model  from  the  point  of  view  of  cost,  return,  and  per¬ 
ceptions. 

5.8.1 .1  Software  Aspects 

QPM  (Quantitative  Project  Management)  and  OPP  (Organization  Process  Performance)  need 
interpretation  and  mapping  vis-a-vis  SQM  and  QPM  (as  in  Software  CMM).  One  issue  at  the 
base  is  how  can  we  find  common/special  causes?  The  root  cause  can  be  at  the  project  or  or¬ 
ganization  level.  While  distinguishing  between  common  and  special  causes,  it  all  comes 
down  to  causal  analysis.  Special  causes  are  addressed  via  corrective  action  and  common  by 
process  reengineering. 

A  lack  of  requirement  at  Level  3  in  CMMI  for  measuring  specific  process  elements  has  been 
observed  and  may  require  a  change  request  (CR).  With  the  change,  with  respect  to  SQM  and 
QPM,  this  will  need  to  be  looked  at  carefully. 

QPM  ACS  may  need  a  correction  in  the  text.  (This  AC  is  written  as  a  goal.) 

How  is  CMMI  addressing  soft  points  of  Software  CMM? 
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5.8.1. 2  Systems  Engineering  Aspects 

One  will  have  to  clearly  understand  system  engineering  work  products,  their  appropriate  es¬ 
timation  models  and  measures. 

What  are  appropriate  measures  for  system  engineering?  (Do  they  measure  artifacts  as  UI  / 
interfaces,  systems,  subsystems  and  work  with  units  such  as  pages  for  requirements  or  re¬ 
quirement  expansion  rate)? 

The  “software  only”  organizations  may  not  understand  some  of  the  aspects  and  hence  an  in¬ 
tegrated  approach  may  remain  missing.  For  other  organizations,  there  may  be  a  need  to  know 
how  to  convert  the  concepts  to  actual  process  requirements. 

5.8.2  The  Transition 

5.8.2.1  Model 

There  is  a  need  to  build  translation  matrix/mechanism  from  Software  CMM  to  CMMI  in 
terms  of  how  KPAs,  practices,  and  subpractices  map  on  to  new  ones.  There  is  also  a  need  to 
know  what/why  practices  have  been  dropped,  added,  or  changed  and  if  there  is  any  terminol¬ 
ogy  difference.  If  this  reflects  improvements  to  engineering  discipline  or  a  consolidation  of 
learnings  from  Software  CMM,  it  needs  to  be  available  in  some  documents  so  that  one  under¬ 
stands  the  basis  for  change. 

Use  of  the  term  “tailoring”  needs  to  be  well  described.  It  was  possible  to  indicate  a  KPA  such 
as  SSM  as  “not  applicable”  if  an  organization  was  not  performing  it.  But  if  SE  subpractices 
are  intermixed,  then  SW-only  organizations  may  need  to  be  able  to  tailor  more  extensively 
than  in  the  Software  CMM.  These  need  to  be  analyzed,  perhaps,  based  on  pilot  experience. 

If  there  is  any  information  on  business  benefits  of  CMMI  or  difference  in  ROI  from  Software 
CMM,  it  needs  to  be  shared  so  that  transition  to  CMMI  can  be  justified. 

5.8.2.2  information 

Some  documentation  on  why  from  Software  CMM  to  CMMI  is  needed  for  SW-only  organi¬ 
zations  that  are  working  fine  with  Software  CMM,  since  their  investment  in  Software  CMM 
training  /  understanding  is  at  risk. 

There  needs  to  be  documents  explaining  what  exceptions  or  extra  processes  a  SW-only  or¬ 
ganization  will  need  due  to  SE-related  model  elements,  and  how  it  will  use  tailoring  or  ex¬ 
ceptions.  The  same  will  have  to  be  well  explained  in  Lead  Assessors  training. 

People  understand  the  model  as  in  Software  CMM.  In  that  context  the  mapping  documents  to 
CMMI  need  to  be  available. 
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It  has  been  mentioned  that  some  information  regarding  considerations  mentioned  at  the 
SEPG  conference  2001  need  to  be  checked  out. 

5.8.3  Recommendations  for  High  Maturity  Organizations 

In  high  maturity  psyche  it  is  natural  to  apply  Level  4  or  5  processes  to  both  project  and  or¬ 
ganization  levels.  They  keep  feeding  each  other  in  a  cycle  (e.g.,  QPM,  DP,  and  PCM).  One 
continuously  looks  at  new  ways  to  improve.  So  one  can  utilize  the  positive  environment. 

One  has  to  see  how  to  protect  the  investment  in  Software  CMM,  not  to  lose  the  momentum  of 
continuous  improvement  and  how  to  retain  the  confidence  of  professionals  in  the  ongoing 
activity.  This  has  a  relation  with  reassessment  time  frame  to  be  looked  at  by  the  organization 
and  if  CMMI  can  be  looked  at  as  a  new  version  of  the  Software  CMM  etc,  with  only  incre¬ 
mental  change. 

One  has  to  see  the  value  of  bringing  in  the  new  CMMI  model  from  management  point  of 
view,  any  extra  investment,  risks  and  evaluation  with  other  competing  or  collaborating  mod¬ 
els  (ISO  9001:2000  or  Six  Sigma,  European  Quality  Model)  and  the  role  these  can  play. 

The  decision  to  move  to  CMMI  will  involve  tailoring  and  a  number  of  interpretations  that 
have  no  precedence.  Clarifications  may  be  sought  directly  from  the  SEI  (and,  perhaps,  not 
from  the  individual  Lead  Assessor)  long  before  one  really  embarks  on  the  move  to  CMMI. 

The  issue  is  how  can  we  find  common/special  causes?  Root  cause  can  be  at  the  project  or 
organization  level?  Why  distinguish  between  common  or  special  causes?  All  comes  down  to 
causal  analysis.  Special  causes  address  via  corrective  action  and  common  by  process  reengi¬ 
neering. 

The  causal  analysis  is  all  integrated,  but  you  have  to  determine  what  type  of  analysis  to  per¬ 
form  and  what  to  do  with  the  results. 

5.8.4  Recommendations  for  the  SEI 

The  current  reference  point  for  organizations  is  Software  CMM  and  CMMI.  They  should  not 
be  expected  to  work  backward  from  CMMI.  The  onus  of  explaining  the  change  is  on  CMMI, 
because  there  is  a  need  to  know  what/why  practices  have  been  dropped,  added,  or  changed 
and  the  best  practice  concept  or  data  behind  it,  not  just  the  analytical  approach. 

There  is  concern  regarding  how  much  measurement  is  necessary.  What’s  enough?  Is  it  for 
those  critical  few  that  impact  business?  Can  that  measurement  be  used  for  decision  making? 
How  far  does  one  have  to  go  with  SPC  or  Quantitative?  Has  real  life  experience  been  used  to 
create  the  model?  If  yes,  does  it  correlate  the  measurements  to  other  project  success  measures 
and  accurate  predictors  of  success? 
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For  a  “SW-only”  scenario,  there  may  not  be  much  published  about  tailoring,  and  the  CMMI 
assessment  may  ask,  Did  you  evaluate  process  change  quantitatively?  Have  you  measured  the 
benefit  of  CMMI  vis-a-vis  existing  Software  CMM?  Have  you  piloted  the  new  process?  If 
not,  this  may  raise  questions  on  the  reliability  of  the  assessment  for  SW-only  organizations. 

Sunset  has  been  planned  to  withdraw  support  after  12/2003  by  not  making  changes,  not  giv¬ 
ing  courses,  no  further  Lead  Assessor  training,  and  no  analyses  on  data  received.  How  Soft¬ 
ware  CMM  can  be  supported  and  taken  further  for  SW-only  organizations  should  be  ad¬ 
dressed. 

Software  CMM  Level  3  organizations  that  are  likely  to  move  to  higher  levels  will  wonder 
what  to  do.  Maybe,  up  to  Level  3,  the  options  of  commonality  in  two  models  for  assessment 
should  be  looked  at. 

High  maturity  organizations  will  have  to  explain  to  customers  why  to  sustain  Software  CMM 
if  the  SEI  does  not  support  it  any  more.  The  date  of  sunset  needs  to  consider  ways  to  give 
choices  and  not  an  impression  of  cutoff. 

5.8.5  Summary 

Process  model  approach  is  measurement-decision-result-belief  as  well  as  process-belief- 
decision-result-measure.  What  is  involved  in  transitioning  to  CMMI  are  people,  culture,  and 
new  approaches  with  little  data.  High  maturity  organizations  work  with  all  simultaneously.  A 
“popular  model”  that  is  widely  used  is  a  suitable  vehicle  to  take  this  forward.  Software  CMM 
is  one  of  the  most  successful  SW  benchmarks  and  change  needs  to  avoid  disconnect. 
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6  Recommendations  for  High  Maturity 
Organizations 


6.1  Recommendations  for  High  Maturity 
Organizations  from  the  1998  Workshop 

The  following  37  recommendations  for  high  maturity  organizations  were  made  by  the  work¬ 
ing  groups  in  the  November  1998  workshop.  They  are  summarized  below,  but  no  status  is 

reported  since  they  were  for  the  community  to  act  on. 

From  the  statistics  working  group: 

1.  Level  4  organizations  should  provide  the  project  teams  with  sufficient  training,  tools, 
and  mentoring  in  statistical  and/or  modeling  methods  to  apply  appropriate  quantitative 
management  techniques  effectively. 

2.  Data  collection  should  be  frequent  enough  to  provide  real-time  control  of  the  process. 

3.  Understanding  variation  is  required  at  Level  4,  but  not  the  use  of  control  charts.  A  high 
maturity  organization  should  choose  the  appropriate  statistical  or  modeling  technique  to 
answer  the  specific  questions  that  it  has. 

4.  A  Level  4  organization  is  not  required  to  have  capable  processes,  but  it  must  understand 
the  capability  of  its  processes. 

5.  Continue  disaggregating  process  control  data  until  the  chart  is  usable. 

6.  The  use  of  SPC  in  earlier  phases  of  the  life  cycle,  such  as  requirements  analysis,  should 
be  institutionalized. 


From  the  measurement  working  group: 

7.  For  data  cost  and  quality,  enforce  the  idea  that  staff  who  generate  data  should  get  to  use 
it. 

8.  For  data  cost  and  quality,  model  the  behavior  to  have  projects  use  data  by  having  meas¬ 
urement  group  work  with  projects. 

9.  For  data  cost  and  quality,  keep  the  linkage  to  use  and  goals. 

10.  For  data  cost  and  quality,  keep  clear  whether  measures  are  for  the  enterprise  or  for  the 
process  only. 

11.  For  data  cost  and  quality,  have  statisticians  and  practitioners  collaborate  on  defect  and 
effort  analysis. 

12.  For  data  cost  and  quality,  define  analysis  at  same  time  as  defining  measures. 
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13.  For  data  cost  and  quality,  have  clear  process  definitions  and  define  measures  as  part  of 
defining  processes. 

14.  For  data  cost  and  quality,  have  management  pull  for  need  for  data. 

15.  For  data  cost  and  quality,  have  common  measurement  criteria. 

16.  For  data  cost  and  quality,  do  not  fragment  collection  and  use  completely. 

17.  For  data  cost  and  quality,  do  not  enter  the  same  data  multiple  times. 

18.  For  data  cost  and  quality,  do  not  bite  off  more  than  you  can  chew. 

19.  For  creation  and  use  of  capability  baselines,  explore  data  to  understand  points  associated 
with  special  causes  of  variation. 

20.  For  creation  and  use  of  capability  baselines,  let  downstream  processes  (e.g.,  testing)  set 
specifications  for  upstream  processes  (e.g.,  defect  detection  activities  such  as  inspec¬ 
tions). 

21 .  For  tying  software  quality  to  business  objectives,  define  process  requirements  like  you 
would  a  product  requirement. 

22.  For  tying  software  quality  to  business  objectives,  look  at  processes  that  are  involved  and 
audience  to  identify  measures. 

23.  For  tying  software  quality  to  business  objectives,  look  at  products  that  are  result  of  proc¬ 
ess  to  identify  entities  to  measure. 

24.  For  tying  software  quality  to  business  objectives,  identify  critical  dimension  of  require¬ 
ment  (cost,  schedule,  quality)  as  attributes  for  measurement. 


From  the  technology  transition  working  group: 

25.  Select  pilot  teams  based  on  technology  adoption  curve  (early  adopters  preferred). 

26.  Consider  “line  of  sight”  coupling  of  technology  objectives  to  organization  and  individ¬ 
ual  objectives,  and  budgeting  and  statusing. 

27.  Establish  a  separate  group  looking  externally  for  “good  matches”  -  NOT  on  project! 

28.  Establish  widespread  publication/briefing  of  tactical  and  strategic  business  and  technol¬ 
ogy  plans. 

29.  Make  the  proposer  the  owner  for  implementation. 

30.  Establish  technology  architecture  for  SPI  support  early;  deploy  supporting  technologies 
“just  in  time”  with  process  deployment. 

31.  Do  “stop  the  world”  training  for  pilot  teams  and  explicit  coaching/mentoring. 

32.  Note  that  mentor  does  NOT  equal  owner  of  the  adoption  (these  are,  however,  comple¬ 
mentary). 

33.  For  increased  sharing  of  technology  deployment  process  with  stakeholders  throughout 
the  TCM  process. . .  communicate. . .  communicate. 

34.  Add  mentors/coaches  from  process  group  and  budget  as  “extra”  staff  to  help  pilots  be 
successful. 

35.  Do  establishing/employing  feedback/refinement  of  whole  product. 
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36.  Recognize  that  first  iteration  does  NOT  equal  the  last  iteration. 

37.  Identify  the  owner  of  the  “sustaining”  part  of  technology  explicitly. 

6.2  Summary  of  2001  Recommendations  for  High 
Maturity  Organizations 

The  following  34  recommendations  for  high  maturity  organizations  were  made  by  the  work¬ 
ing  groups  in  the  March  2001  workshop.  They  are  only  summarized  below  (some  re-writing 
was  necessary  to  summarize  the  recommendations);  more  detailed  discussions  may  be  found 
in  the  above  section  on  working  group  reports. 

1 .  From  the  measurement  working  group:  Involve  those  impacted  by  the  measurement  pro¬ 
gram  is  -  do  it  with  them,  not  to  them. 

2.  Start  with  small,  focused  efforts  to  generate  some  early  successes. 

3.  Integrate  project  measures  and  business  objectives. 

4.  Revisit  the  basics  and  review  the  purpose  and  need  for  each  measure. 

5.  Automate  collection  and  analysis  as  much  as  practicably  possible. 

6.  Address  change  management  as  applied  to  measurement  in  the  context  of  high  maturity. 

From  the  Six  Sigma  working  group: 

7.  Gain  top-level  management  support.  Show  how  Six  Sigma  can  be  used  as  tool  and  the 
“next  step”  after  reaching  Software  CMM  Level  5. 

8.  Use  Six  Sigma  to  increase  practitioner  ownership,  to  give  the  practitioners  a  new  tool,  to 
generate  new  interest,  to  increase  process  agility. 

9.  When  implementing  Six  Sigma,  place  focus  on  training  and  piloting  with  one  project 
and  one  problem. 

10.  If  currently  using  Six  Sigma,  publish  experiences  and  results. 


From  the  CMMI  working  groups: 

1 1.  Use  valuable  lessons  learned  from  high  maturity  organizations.  Implement  measure¬ 
ments  early,  set  up  an  engineering  process  group,  peer  reviews  and  other  forms  of  verifi¬ 
cation,  process  improvement  adoption  lessons  learned. 

12.  Follow  TQM  principles  during  strategic  planning;  identify  marketing  areas  and  opera¬ 
tional  direction.  If  you  have  an  initiative  that  crosses  the  organization  (i.e.,  establishes  an 
“umbrella”),  it  becomes  easier  to  deploy  CMMI  due  to  a  common  framework. 

13.  Industry  has  to  examine  territory  beyond  Level  5.  Protect  the  investment  in  Software 
CMM,  not  to  lose  the  momentum  of  continuous  improvement  and  how  to  retain  the  con¬ 
fidence  of  professionals  in  the  ongoing  activity. 

15.  See  the  value  of  bringing  in  the  new  CMMI  model  from  management  point  of  view. 

16.  Seek  clarifications  directly  from  the  SEI  before  embarking  on  the  move  to  CMMI. 
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From  the  statistics  working  group: 

17.  Get  competent  statistical  guidance  with  experience  beyond  manufacturing, 

18.  Let  the  data  and  objectives  determine  the  statistical  methods  used. 

19.  Set  quantitative  objectives  tied  to  business  goals. 

20.  Simplify  the  presentation  of  statistical  results. 

21.  Use  data  to  gain  understanding  and  control  (Level  4)  and  guide  improvement  (Level  5). 

22.  Learn  about  Six  Sigma  and  the  tools  it  offers  for  Level  4  and  5  activities. 

From  the  change  management  working  group: 

23.  Work  force  management  processes  must  mature  commensurate  with  other  disciplines. 

24.  Use  broad  range  of  sources  for  improvement  ideas. 

25.  Explore  how  to  do  “cultural  due  diligence”  to  handle  the  pressure  and  risk  of  acquisi¬ 
tions  and  mergers. 

26.  Establish  resources  and  energy  for  process  improvement. 

27.  If  an  organization  asks  its  people  to  participate  in  change  activities,  it  should  make  it 
easy  by  providing  appropriate  support  (tool  support,  etc.)  and  make  sure  it  provides  a 
benefit  to  the  people. 

From  the  Internet  speed  working  group: 

28.  Participate  in  SEI  research  in  process  agility  and  Internet  speed. 

29.  Publish  and  share  learnings  on  agile  high  maturity  organizations. 

30.  Contribute  to  a  common  database  (industry  wide). 

3 1 .  Encourage  the  community  to  make  data  available  to  the  SEIR. 

32.  Keep  an  open  mind,  evaluate  new  trends,  and  propose  changes  as  relevant  to  the  SEI. 

33.  Balance  rigor  with  agility  when  defining  project  processes. 

34.  Gather  and  use  data  to  understand  the  risks  and  consequences  that  are  associated  with 
agile  processes. 
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7  Summary  of  Recommendations  for  the 
SEI 


7.1  Recommendations  for  the  SEI  and  Status  - 
November  1998  Workshop 

The  following  25  recommendations  for  the  SEI  were  made  by  the  working  groups  in  the 
1998  workshop.  Status  with  respect  to  SEI  actions  that  have  been  taken  on  these  recommen¬ 
dations  is  included. 

From  the  statistics  working  group: 

1.  The  SEI  should  clarify  to  the  SPI  community  whether  the  current  operating  model  is 
Software  CMM  Version  1.1  as  written;  Version  1.1  as  reinterpreted,  clarified,  and  elabo¬ 
rated  in  Software  CMM  Version  2  Draft  C;  or  what  we  wish  we  had  said  when  we  wrote 
Version  1.1,  which  is  not  exactly  what  we  said. 

Status:  Accepted,  Version  LI  as  written  is  the  current  operating  model  of  the  Software 
CMM. 

2.  The  SEI  should  clarify  confusing  high  maturity  issues  in  the  CMMI  model  for  Level  4  at 
the  goal  and  practice  level. 

Status:  Accepted.  Interpretation  issues  for  Software  CMM  vl.  1  are  addressed  in  the 
High  Maturity  with  Statistics  course.  The  emphasis  is  on  the  terminology  used  in  vl.l 
and  what  it  means. 

3.  The  SEI  should  maintain  flexibility  in  the  range  of  quantitative  methods  that  are  legiti¬ 
mate  (or  required)  at  Level  4.  Many  different  quantitative  methods  can  be  used  to  sup¬ 
port  quantitative  management. 

Status:  Accepted.  The  High  Maturity  with  Statistics  course  suggests  that  a  number  of 
statistical  and  quantitative  techniques  can  be  effective.  Some  recommendations,  e.g.„ 
XmR  charts,  are  made  for  those  who  wish  a  starting  point  to  consider. 

4.  The  SEI  should  get  input  from  more  organizations  in  building  consensus  on  high  matur¬ 
ity  practices  and  disseminate  this  information  for  review  and  guidance  to  organizations 
that  need  its  guidance. 

Status:  Accepted.  The  High  Maturity  Workshop  is  becoming  a  regular  event,  scheduled 
on  roughly  an  18-month  timeframe.  Panels  on  SPC,  what  it  means  to  be  Level  4,  what  it 
means  to  be  Level  5,  etc.,  that  involve  SEI  staff.  Lead  Assessors,  and  representatives 
from  high  maturity  organizations  have  been  held  at  the  various  SEPG  conferences 
around  the  world. 

5.  The  SEI  should  publish  a  compendium  of  quantitative  management  practices  (including 
examples  other  than  SPC)  currently  in  use  and  their  benefits. 
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Status:  Accepted  The  High  Maturity  with  Statistics  course  contains  the  beginning  of 
such  a  compendium. 

6.  The  SEI  should  request  Lead  Assessors  to  supply  case  studies  on  Level  4  and  5  organi¬ 
zations. 

Status:  Accepted.  Position  papers  from  Lead  Assessors  and  Evaluators  were  solicited  for 
this  report,  with  limited  success.  Lead  Assessors  have  been  encouraged  to  present  case 
studies  at  various  conferences  and  journals. 

7.  The  SEI  should  create  guidelines  for  applying  quantitative  management  techniques 
based  on  industry  lessons  learned. 

Status:  Accepted.  This  will  probably  be  a  technical  report. 

8.  The  SEI  should  not  be  the  final  authority  on  statistics. 

Status:  Accepted.  The  SEI  is,  however,  the  authority  on  interpreting  the  Software  CMM, 
which  is  really  the  intent  of  this  point.  The  real  problem  is  that  the  guidance  on  inter¬ 
preting  the  model  has  been  inadequate,  although  steps  are  being  taken,  such  as  develop¬ 
ing  the  High  Maturity  with  Statistics  course,  to  address  this  problem. 

As  was  discussed  in  the  report  on  the  1999  workshop,  this  point  was  inspired  by  a  dis¬ 
cussion  of  2-sigma  versus  3-sigma  control  limits.  Some  high  maturity  organizations  are 
using  2-sigma  limits  to  trigger  action,  and  it  was  observed  that  several  SEI  statistics  ex¬ 
perts  have  commented  that  these  are  not  valid  control  limits.  Paulk  stated  that,  in  his 
judgment,  action  limits''  based  on  2-sigma  were  legitimately  based  on  an  understand- 
of  variation  and  thus  could  be  considered  a  valid  quantitative  management  tech¬ 
nique  at  Level  4.  Some  statisticians  consider  these  to  be  valid  control  limits;  other  stat¬ 
isticians  note  that  this  is  an  explicit  violation  of  Shewhart's  rationale  for  choosing  3- 
sigma  limits  and  state  that  2-sigma  limits  are  incorrect.  Given  the  rift  in  the  statistical 
community  on  this  issue,  anyone  using  2 -sigma  limits  should  be  educated  that  this  is  not 
generally  accepted  practice,  and  the  use  of  the  term  ''2-sigma  control  limits, "  unless 
made  in  ignorance,  is  likely  to  be  considered  as  taking  a  position  in  a  heated  debate  that 
makes  the  proponent  fair  game.  It  is  fair  to  say  that  most  SEI  staff  knowledgeable  in 
SPC  are  aligned  with  Don  Wheeler's  philosophies,  so  characterizing  2-sigma  limits  as 
control  limits  is  likely  to  lead  to  a  correction  in  terminology. 

In  general,  the  SEI  acknowledges  that  there  are  many  sources  of  statistical  expertise, 
and  not  all  of  those  sources  are  in  perfect  accord.  While  the  SEI  will  continue  to  act  as 
an  arbiter  of  unreasonable  interpretations,  it  is  accepted  that  some  quantitative  and  sta¬ 
tistical  questions  remain  inherently  controversial,  and  a  wide  latitude  of  implementa¬ 
tions  will  advance  the  understanding  of  our  community  of  what  techniques  are  effective 
over  time. 


From  the  assessment  working  group: 

9.  Re-establish  the  CMM  Advisory  Board. 

Status:  Rejected.  The  CMMI  Steering  Group  fulfills  this  function  for  CMMI,  which  will 
replace  the  Software  CMM  in  2003. 

10.  Establish  mandatory  supplemental  training  of  any  Lead  Assessor  (to  lead  Level  4  and  5 
assessments). 

Status:  Under  consideration.  Implementation  will  probably  be  in  the  context  of  CMMI. 
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11.  Gather  data  regarding  the  “problem  of  inconsistent  results  at  Level  4  and  5.” 

Status:  Accepted. 

12.  Strengthen  QA  provisions  for  Lead  Assessors. 

Status:  Accepted.  Will  Hayes  is  the  contact  person  for  further  information  in  this  area. 

13.  Periodically  conduct  High  Maturity  Workshops. 

Status:  Accepted.  The  High  Maturity  Workshop  is  planned  to  be  held  on  roughly  an  18- 
month  cycle. 

14.  Elicit  papers  from  the  community  at  large. 

Status:  Accepted. 

15.  Identify  criteria  for  qualified  referees. 

Status:  Accepted.  For  the  SEPG  Conferences,  a  relatively  small  pool  of  qualified  refe¬ 
rees  has  evolved.  For  further  information,  contact  Mark  Paulk  at  the  SEL 

16.  Add  Report  of  the  Workshop  Proceedings  and  mandate  to  grow  further. 

Status:  Overcome  by  events.  Specific  to  a  draft  document  reviewed  at  the  workshop, 
“SE/  Strategy  for  Ensuring  Valid  Implementation  and  Appraisal  of  Level  4  and  Level  5 
Process  Areas-October  28,  1999,  ”  which  has  not  been  released. 

17.  Drop  the  word  “informal”  from  title  of  paragraph  8.  Re-title  “Communications  between 
the  SEI  and  the  CMM  User  Community.” 

Status:  Overcome  by  events.  Specific  to  a  draft  document  reviewed  at  the  workshop, 
''SEI  Strategy  for  Ensuring  Valid  Implementation  and  Appraisal  of  Level  4  and  Level  5 
Process  Areas-October  28,  1999,  ”  which  has  not  been  released. 

18.  Make  high  maturity  materials  available  through  the  Transition  Partners. 

Status:  Accepted.  It  is  intended  that  a  CD  with  a  high  maturity  tutorial  will  be  provided 
to  attendees  of  the  High  Maturity  with  Statistics  course,  but  this  CD  has  not  yet  been 
built. 

19.  Provide  high  maturity  training  in  other  geographic  areas  (e.g.,  Middle  East,  India,  Aus¬ 
tralia,  etc.). 

Status:  Accepted.  This  has  been  done  to  a  limited  degree,  primarily  with  the  European 
Software  Process  Improvement  Foundation  in  the  U.K.  A  few  other  offerings  are 
planned  in  Europe  in  the  near  future.  No  plans  currently  exist  for  high  maturity  training 
outside  of  the  U.S.  and  Europe,  but  opportunities  will  be  considered  as  they  arise. 


From  the  CMMI  working  group: 

20.  Pick  a  single  representation,  continuous  or  staged,  for  the  CMMI  model. 
Status:  Rejected. 


From  the  technology  transition  working  group: 

21.  OPF  should  address  integrating  process  architecture  and  technology  support  architec¬ 
ture. 
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Status:  Passed  along  to  the  CMMI  team  for  consideration  in  a  fiitiire  version  of  the 
CMMI  model. 

22.  Include  more  “feasibility”  focus  for  technology — encouraging  organizations  to  master 
analyzing  the  feasibility  of  different  technology  alternatives  earlier. 

Status:  Passed  along  to  the  CMMI  team  for  consideration  in  a  fiiture  version  of  the 
CMMI  model. 

23.  Add  subpractices  /  examples  on  appropriate  technology  support  in  relevant  KPAs  (e.g., 
problem  report  tracking,  defect  tracking). 

Status:  Passed  along  to  the  CMMI  team  for  consideration  in  a  future  version  of  the 
CMMI  model. 

24.  Add  more  front  matter  on  technology  implementation  vs.  TCM  mastery. 

Status:  Passed  along  to  the  CMMI  team  for  consideration  in  a  future  version  of  the 
CMMI  model. 

25.  Emphasize  appropriate  “data  collection”  technology  support  at  Level  2/3  and  above  as  a 
way  to  reduce  barriers  to  process  deployment. 

Status:  Passed  along  to  the  CMMI  team  for  consideration  in  a  future  version  of  the 
CMMI  model. 

7.2  Recommendations  for  the  SEI  and  Status  - 
March  2001  Workshop 

The  following  38  recommendations  for  the  SEI  were  made  by  the  working  groups  in  the 
March  2001  workshop.  They  are  only  summarized  below  (some  re-writing  was  necessary  to 
summarize  the  recommendations);  more  detailed  discussions  may  be  found  in  the  above  sec¬ 
tion  on  working  group  reports. 

From  the  measurement  working  group: 

1.  Encourage  organizations’  understanding  of  measurement  at  lower  maturity  levels. 

Status:  Accepted.  This  is  addressed  by  the  Measurement  and  Analysis  process  area  at 
Level  2  in  the  CMMI  model  (staged  representation). 

2.  Address  change  management  as  applied  to  measurement  in  the  context  of  high  maturity. 

Status:  Passed  along  to  the  CMMI  team  for  consideration  in  a  future  version  of  the 
CMMI  model. 

3.  Provide  guidelines  and  examples  for  quantitative  analysis  techniques  and  methods  that 
are  acceptable  at  Level  4  (other  than  SPC). 

Status:  Accepted.  Under  consideration  for  papers  and  revisions  to  the  High  Maturity 
with  Statistics  course. 
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From  the  assessment  working  group: 

4.  Encourage  Lead  Assessors  and  Lead  Evaluators  to  take  additional  training  in  high  ma¬ 
turity  topics. 

Status:  Accepted.  The  High  Maturity  with  Statistics  and  SPC  for  Software  courses  are 
the  recommended  training  at  this  time. 

5.  Suggest  appraisers  use  a  standard  experience  /  qualification  matrix  as  a  means  for  Lead 
Assessors  and  Evaluators  to  describe  their  level  and  scope  of  experience  (maturity  lev¬ 
els,  disciplines,  etc.)  and  for  appraisal  customers  to  better  match  LA  /  LE  qualifications 
to  the  parameters  of  their  appraisal. 

Status:  Passed  along  to  the  CMMl  team  for  consideration  in  a  future  version  of  the 
SCAMPI  method. 

6.  Provide  additional  model  and  method  implementation  guidance  for  appraisers. 

Status:  Accepted.  The  High  Maturity  with  Statistics  course  embodies  this  guidance.  Pa¬ 
pers  will  be  published  as  they  are  completed. 

7.  Utilize  newly  available  mechanisms  for  disseminating  this  guidance  to  the  Lead  Ap¬ 
praiser  community  (for  example,  the  new  SEI  Lead  Assessor  web  site,  the  International 
Association  of  Professional  Lead  Assessors  [lAPLA]  Web  site). 

Status:  Accepted. 


From  the  Six  Sigma  working  group: 

8.  Retain  the  Software  CMM  and  the  focus  it  brings  to  software. 

Status:  Rejected.  CMMl  contains  the  software  model  that  will  replace  the  Software 
CMM  in  2003. 

9.  Explore  including  Six  Sigma  in  future  versions  of  the  Software  CMM  as  a  tool  to  aid 
Level  4  and  Level  5  organizations  in  continuous  and  measurable  improvement. 

Status:  Passed  along  to  the  CMMl  team  for  consideration  in  a  future  version  of  the 
CMMl  model. 

10.  Recognize  that  Six  Sigma  is  not  standardized,  that  it  has  a  wide  spectrum  of  tools.  Focus 
on  referring  the  relevant  tools  from  the  Six  Sigma  assortment. 

Status:  Accepted. 

11.  Publicly  recognize  Six  Sigma  as  a  useful  tool  for  software  organizations.  Publish  articles 
in  publications  such  as  CrossTalk. 

Status:  Accepted. 

12.  Show  the  relation  between  the  different  models  and  frameworks.  Show  the  connections, 
the  mappings,  the  benefits.  Describe  the  broad  toolset  necessary  to  develop  a  sound 
quality  management  system. 

Status:  Passed  along  to  the  CMMl  team  for  consideration  in  a  future  version  of  the 
CMMl  model. 

13.  Develop  SEI  courses  on  the  application  of  Six  Sigma  to  software  organizations. 


CMU/SEI-2001-SR-014 


181 


Status:  Accepted.  This  will  probably  be  done  within  the  context  of  the  High  Maturity 
with  Statistics  course  as  it  continues  to  be  refined. 


From  the  CMMI  working  groups: 

14.  Why  is  CMMI  not  considered  Software  CMM  Version  3.0?  Why  is  there  not  a  logical 
progression  from  Software  CMM  to  CMMI  (in  models,  training,  and  assessment  meth¬ 
ods)? 

Status:  Passed  along  to  the  CMMI  team  for  consideration. 

15.  Develop  a  CMMI  time-bound  release  plan  for  industry  involving  all  aspects  of  the  or¬ 
ganizations  (e.g.,  marketing,  human  resources,  etc.). 

Status:  Passed  along  to  the  CMMI  team  for  consideration. 

16.  Capture  the  lessons  learned  from  high  maturity  organizations  in  how  to  quickly  and  in¬ 
telligently  implement  continuous  process  improvement  and  provide  industry  a  road  map 
to  get  through  the  model. 

Status:  Passed  along  to  the  CMMI  team  for  consideration.  The  High  Maturity  with  Sta¬ 
tistics  course  is  largely  model  independent  and  may  address  this  concern.  PSP  and  TSP 
are  also  likely  to  be  helpful. 

17.  Establish  industry-wide  support  and  buy-in  for  transition  to  CMMI,  including  involve¬ 
ment  from  the  commercial  sector.  Ensure  that  software-only  organizations  can  see  that 
the  model  works  for  them  too  and  that  there  are  clear  guidelines  of  the  model  for  appli¬ 
cation  to  software-only  organizations. 

Status:  Passed  along  to  the  CMMI  team  for  consideration. 

18.  Explain  the  changes  in  CMMI  because  there  is  a  need  to  know  what  /  why  practices 
have  been  dropped,  added,  changed  and  the  best  practice  concept  or  data  behind  it,  not 
just  analytical  approach. 

Status:  Passed  along  to  the  CMMI  team  for  consideration. 

19.  Clarify  how  much  measurement  is  enough  for  CMMI.  How  far  does  one  have  to  go  with 
SPC  or  quantitative? 

Status:  Passed  along  to  the  CMMI  team  for  consideration. 

20.  Provide  quantitative  information  on  the  benefit  of  CMMI  vis-a-vis  existing  Software 
CMM. 

Status:  Passed  along  to  the  CMMI  team  for  consideration. 

21.  Clarify  how  the  Software  CMM  can  be  supported  after  the  2003  sunset  of  Software 
CMM  vl.l  and  taken  further  for  software-only  organizations. 

Status:  Passed  along  to  the  CMMI  team  for  consideration.  A  position  paper  has  been 
drafted  and  is  currently  under  review. 

22.  Consider  options  of  commonality  in  the  two  models  for  assessment,  especially  at  Levels 
3  and  higher. 

Status:  Passed  along  to  the  CMMI  team  for  consideration. 

23.  Explain  the  date  of  sunset  in  ways  to  give  choices  and  not  an  impression  of  cut  off. 
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Status:  Passed  along  to  the  CM  MI  team  for  consideration.  A  position  paper  has  been 
drafted  and  is  currently  under  review. 


From  the  statistical  techniques  working  group: 

24.  Improve  Software  CMM  /  CMMI  to  focus  on  implementing  quantitative  management 
rather  than  focusing  too  narrowly  on  a  specific  method  of  implementation  (i.e.,  SPC). 

Status:  Passed  along  to  the  CMMI  team  for  consideration.  The  High  Maturity  with  Sta¬ 
tistics  course  attempts  to  balance  these  issues. 

25.  Perform  research  and  offer  training  on  quantitative  methods  in  addition  to  SPC. 

Status:  Accepted.  A  number  of  measurement  courses  and  quantitative  research  are  being 
performed  relative  to  process  improvement  at  the  SEI,  primarily  by  the  Software  Engi¬ 
neering  Measurement  and  Analysis  project. 

26.  Continue  to  capture  and  disseminate  community  experience. 

Status:  Accepted. 

27.  Encourage  Level  4  and  5  organizations  to  publish  their  results  and  insights. 

Status:  Accepted. 

28.  Provide  better  guidance  and  training  for  Lead  Assessors. 

Status:  Accepted.  The  High  Maturity  with  Statistics  and  SPC  for  Software  courses  at¬ 
tempt  to  address  this  concern. 


From  the  change  management  and  people/cultural  issues  working  group: 

29.  Consider  hosting  an  explicit,  ongoing,  invitation-only  forum  targeted  to  senior  manage¬ 
ment,  for  example,  “a  sponsor  workshop”  to  cover  roles  and  skills  for  sponsors. 

Status:  Passed  along  to  SEI  management  for  further  consideration. 

30.  There  should  be  a  “What  does  management  look  like  in  high  maturity  organizations?” 
course. 

Status:  Rejected.  We  will  consider  adding  material  on  this  topic  to  the  High  Maturity 
with  Statistics  course  and/or  incorporating  it  into  other  SEI  courses. 

31.  A  group  from  this  workshop  should  work  with  the  SEI  to  capture  best  practices  in 
change  management  because  we  heard  considerable  consistency  in  what  works,  and  this 
common,  best  practice  could  be  captured  and  disseminated  throughout  the  community. 
Several  participants  from  this  workshop  have  volunteered  to  participate  in  preparing  a 
best  practices  guide  for  change  management.  The  SEI  should  consider  publishing  this 
guide. 

Status:  Accepted.  The  primary  means  we  have  been  using  are  the  high  maturity  work¬ 
shop  and  survey  reports,  plus  papers  and  panels  at  various  conferences  (primarily  the 
various  SEPG  Conferences).  We  will  try  to  be  more  proactive  in  addressing  change 
management  in  collaboration  with  our  industry  colleagues. 

32.  Define  and  support  reasonable  expectations  of  Level  4  and  5  organizations — better  edu¬ 
cate  customers  on  concepts  of  process  capability  vs.  process  performance. 
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Status:  Accepted.  We  have  been  building  a  consensus  on  what  the  expectations  should  he 
via  panels  and  presentations  at  various  conferences,  as  well  as  the  workshops  and  sur¬ 
veys.  The  mechanism  is  likely  to  be  in  the  form  of  published  papers. 

33.  The  SEI  should  more  consistently  consider  global  issues.  For  example,  consider  holding 
the  next  high  maturity  workshop  in  Bangalore. 

Status:  Accepted.  Sally  Cunningham  has  been  appointed  as  the  SETs  Director  of  Inter¬ 
national  Relations.  She  will  be  apprised  of  these  and  related  concerns  as  we  become 
aware  of  them  and  work  with  our  sponsor  and  the  community  to  address  them. 

34.  The  SEI  might  try  to  become  a  high  maturity  organization  or  provide  more  working  ex¬ 
amples  of  high  maturity  organizations. 

Status:  Accepted.  We  have  been,  and  will  continue  to  be,  strong  advocates  of  case  stud¬ 
ies  of  process  improvement  and  high  maturity  practices.  We  will  continue  to  encourage 
this  kind  of  information  exchange. 


From  the  Internet  speed  working  group: 

35.  Provide  a  better  definition  of  “light”  if  this  topic  is  intended  to  be  studied. 

Status:  Accepted.  We  are  moving  to  the  '‘agile  methodologies*'  terminology  recently  es¬ 
poused  by  the  Agile  Alliance.  See  http://www.agilealliance.org/for  further  information. 

36.  Sponsor  a  study  of  high  maturity  organizations  that  are  doing  Internet  development  pro¬ 
jects. 

Status:  Under  consideration.  This  could  be  folded  into  the  next  high  maturity  survey, 
which  will  probably  be  in  2003. 

37.  Sponsor  follow-on  work  to  existing  exploratory  study  on  Internet  speed. 

Status:  Under  consideration.  Contact  Linda  Levine  for  further  information. 

38.  Sponsor  a  user  group  (high  maturity  organizations)  to  consider  Internet  speed  and  other 
topics. 

Status:  Under  consideration.  Contact  Linda  Levine  for  further  information. 
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8  Conclusions 


The  March  2001  High  Maturity  Workshop  discussed  a  number  of  important  issues  to  the 
software  community  and  the  SEI.  Although  there  was  less  participation  than  anticipated,  this 
was  at  least  partially  due  to  insufficient  advance  notice  that  the  workshop  would  be  held.  Ten¬ 
tative  plans  are  to  hold  another  High  Maturity  Workshop  in  the  second  week  of  November  in 
2002. 
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/UnderstandHM.pdf>. 
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Appendix  A:  List  of  Workshop  Participants 


Julie  Barnard 
Roger  Bate 
Bruce  Boyd 
Kelley  Butler 
Anita  Carleton 
Lynn  Carter 
Mary  Beth  Chrissis 
Bill  Curtis 
Anthony  D’ Souza 
Donna  Dunaway 
Eileen  Forrester 
Suzanne  Garcia 
Diane  Gibson 
Dennis  Goldenson 
Vivek  Govilkar 
Asha  Goyal 
Subrata  Guha 
William  Hayes 


United  Space  Alliance,  Houston,  TX 

Software  Engineering  Institute,  Pittsburgh,  PA 

The  Boeing  Company,  St.  Louis,  MO 

US  Air  Force,  Tinker  AFB,  OK 

Software  Engineering  Institute,  Pittsburgh,  PA 

Software  Engineering  Institute,  Pittsburgh,  PA 

Software  Engineering  Institute,  Pittsburgh,  PA 

TeraQuest  Metrics,  Austin,  TX 

Satyam  Computer  Services  Limited,  India 

Software  Engineering  Institute,  Pittsburgh,  PA 

Software  Engineering  Institute,  Pittsburgh,  PA 

Software  Engineering  Institute,  Pittsburgh,  PA 

Carnegie  Mellon  University,  Pittsburgh,  PA 

Software  Engineering  Institute,  Pittsburgh,  PA 

i-flex  solutions  Ltd.,  Mumbai,  India 

IBM  Global  Services,  New  Delhi,  India 

Satyam  Computer  Services  Limited,  Charlottesville,  VA 

Software  Engineering  Institute,  Pittsburgh,  PA 
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William  Hefley 

Christian  Hertneck 

Bob  Hoekstra 

Craig  Hollenbach 

John,  Jun  An  Yu 

Gargi  Keeni 

Mike  Konrad 

Linda  Levine 

James  McHale 

Judah  Mogilensky 

Joseph  Morin 

Muthuramalingam  Ra- 
jamanickam 

Gerald  Ourada 

Mark  Paulk 

Mary  Lynn  Penn 

Padmanabhan  Rajasek- 
haran 

Somashekhar 

Ramadevanahalli 

Joan  Romine 

Elizabeth  Schulte 

Jitendra  Shreemali 


Q-Labs,  Pittsburgh,  PA 
Software  Engineering  Institute.  Pittsburgh.  PA 
Philips  Software  Centre  Limited.  Bangalore,  India 
Litton  /  PRC,  McLean,  VA 
Motorola  (China),  Beijing.  China 
Tata  Consultancy  Services,  Calcutta,  India 
Software  Engineering  Institute,  Pittsburgh,  PA 
Software  Engineering  Institute,  Pittsburgh,  PA 
Software  Engineering  Institute,  Pittsburgh,  PA 
Process  Enhancement  Partners,  Inc.,  Silver  Spring,  MD 
Integrated  System  Diagnostics,  Inc.,  Pocasset,  MA 
HCL  Technologies,  Chennai,  India 

Lockheed  Martin  Aeronautics  Co.,  Fort  Worth,  TX 
Software  Engineering  Institute,  Pittsburgh,  PA 
Lockheed  Martin,  Philadelphia,  PA 
Mastek  Ltd.,  Andheri,  India 

CG-Smith  Software  Ltd.,  Bangalore,  India 

The  Boeing  Company,  St.  Louis,  MO 

The  Boeing  Company,  Huntington  Beach,  CA 

Philips  Software  Centre  Limited,  Bangalore,  India 
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Jeannine  Siviy 
Ashok  Sontakke 
Albert  Soule 
Phillip  Sperling 
Wendy  Irion  Talbot 
Jim  Vanfleet 
Mel  Wahlberg 
Gian  Wemyss 
David  White 
David  Zubrow 


Software  Engineering  Institute,  Pittsburgh,  PA 
Zensar  Technologies  LTD,  Pune,  India 
Software  Engineering  Institute,  Pittsburgh,  PA 
Telos-OK,  Lawton,  OK 

Computer  Sciences  Corporation,  Moorestown,  NJ 
US  Air  Force,  Hill  AFB,  UT 
Computer  Sciences  Corporation,  Laurel,  MD 
Software  Engineering  Institute,  Pittsburgh,  PA 
Software  Engineering  Institute,  Pittsburgh,  PA 
Software  Engineering  Institute,  Pittsburgh,  PA 
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Appendix  B:  List  of  High  Maturity 

Organizations  Participating 


The  following  list  of  high  maturity  organizations  lists  most  of  the  known  Level  4  and  5  or¬ 
ganizations.  The  organizations  that  participated  in  the  workshop  are  noted  with  a  V  in  column 
1  of  the  table. 

As  of  May  2001,  the  full  list,  of  which  the  published  list  is  a  subset,  includes  132  high  matur¬ 
ity  organizations,  a  subset  of  which  is  listed  below:  There  are 

•  71  Level  4  organizations 

•  61  Level  5  organizations 

It  is  interesting  to  note  that  74  of  the  high  maturity  organizations  assessed  are  outside  the 
United  States. 

•  Australia:  1  Level  4  organization 

•  China:  2  Level  5  organizations 

•  France:  1  Level  4  organization 

•  India:  30  Level  4  organizations 

•  India:  39  Level  5  organizations 

•  Israel:  1  Level  4  organization 

Of  the  132  known  high  maturity  organizations,  35  participated  in  the  workshop.  In  some  in¬ 
stances,  the  same  person  represented  multiple  organizations,  e.g.,  Gargi  Keeni.  Of  the  35, 
there  were  representatives  from  23  organizations  in  India  and  one  organization  in  China. 

Please  be  aware  of  the  following  issues  regarding  this  list. 

•  The  SEI  does  not  certify  companies  at  maturity  levels. 

•  The  SEI  does  not  confirm  the  accuracy  of  the  maturity  levels  reported  by  the  Lead  Asses¬ 
sors  or  organizations. 

•  This  list  of  Level  4  and  5  organizations  is  by  no  means  exhaustive;  we  know  of  other 
high  maturity  organizations  that  have  chosen  not  to  be  listed. 

•  The  SEI  did  not  use  information  stored  within  its  Process  Appraisal  Information  System 
to  produce  this  document. 
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•  The  organizations  listed  gave  explicit  permission  to  publish  this  information. 

•  No  information  obtained  in  confidence  was  used  to  produce  this  list. 


The  following  information  is  contained  in  this  table,  as  reported  by  the  organization: 

•  Full,  correct  name  of  the  organization  (with  acronyms  defined),  including  city  and 
state  (or  country) 

•  Point  of  contact:  name  and  email  address 

•  Maturity  level  assessed 

•  Month  and  year  of  assessment  (Including  the  form  of  assessment  if  different  from  CBA 
I  PI  with  Lead  Assessor. ) 

•  Lead  Assessor(s)  (Lead  Evaluators  are  annotated  with  LE;  some  appraisers  are  both 
LAs  and  LEs  Some  Lead  Assessors  are  now  inactive  (I)  and  no  longer  listed  on  the  LA 
and  LE  lists.) 


HighM 

aturity  Organization 

Aiitec 

Laval,  France 

Point  of  Contact:  Jerome  Barbierjeb@alitec.net;  Jean  Noel  Martin,  jnm@alitec.net 
Maturity  Level:  4 

Date  of  Appraisal:  July  2000 

Lead  Assessor(s):  Jean- Yves  Le  Goic 

Atos  Origin  India  (formerly  Origin  Information  Technology  India  Limited) 

Mumbai,  India 

Point  of  Contact:  Darayus  Desai,  darayus.desai@atosorigin.com 

Maturity  Level:  5 

Date  of  Appraisal:  Nov  2000 

(CAF-compliant  Process  Professional  Assessment  Method) 

Lead  Assessor(s):  (Cyril  Dyer  -  Compita  Assessor) 

BFL  Software  Limited 

Bangalore,  India 

Point  of  Contact:  Madhukumar  P.S.,  Madhukumar.PS@bflsoftware.com 

Maturity  Level:  4 

Date  of  Appraisal:  June  1999 

Lead  Assessor(s):  Carolyn  Swanson 

Boeing  Company,  Aircraft  &  Missiles  &  Phantom  Works  Southern  California, 

Long  Beach,  CA 

Point  of  Contact:  George  H.  Kasai,  george.h.kasai@boeing.com 

Maturity  Level:  5 

Date  of  Appraisal:  Dec  1997 

Lead  Assessor(s):  Andy  Felschow,  Jeff  Facemire 

Boeing  Company,  Military  Aircraft  &  Missile  Systems  F/A-I8  Mission  Computer 

St.  Louis,  MO 

Point  of  Contact:  Bruce  A.  Boyd,  bruce.a.boyd@boeing.com; 

Robert  L.  Allen,  robert.l.aIlen3@boeing.com 

Maturity  Level:  4 

Date  of  Appraisal:  Nov  1999  (SCE) 

Lead  Assessor(s):  Roy  Queen  (LE),  Jeff  Perdue 

194 


CMU/SEI-2001-SR-014 


High  Maturity  Organization _ 

\  Boeing  Company,  Reusable  Space  Systems  and  Satellite  Programs 

Downey  &  Seal  Beach,  CA 

Point  of  Contact:  Don  Dillehunt,  donald.d.diIlehunt@boeing.com 

Maturity  Level:  5 

Date  of  Appraisal:  Oct  1999 

Lead  Assessor(s):  Andy  Felschow,  Jeff  Facemire _ 

Boeing  Company,  Space  Transportation  Systems 
Kent,  WA 

Point  of  Contact:  Gary  Wigle,  gary.b.wigle@boeing.com 
Maturity  Level:  5 
Date  of  Appraisal:  July  1996 

Lead  Assessor(s):  Steve  Masters,  Mark  Paulk _ 

CG-Smith  Software 
Bangalore,  India 

Point  of  Contact:  G.N.  Raghavendra  Swamy,  raghav@cgs.cgsmith.soft.net 
Maturity  Level:  5 
Date  of  Appraisal:  Sept  1999 

Lead  Assessor(s):  Richard  Storch _ 

Citicorp  Overseas  Software  Limited  (COSL) 

Mumbai,  India 

Point  of  Contact:  Makarand  Khandekar,  makarand.khandekar@citicorp.com 
Maturity  Level:  5 
Date  of  Appraisal:  Oct  1999 
Lead  Assessor(s):  John  Sheckler 


Cognizant  Technology  Solutions 
Chennai,  India 

Point  of  Contact:  Emani  BSP  Sarathy,  esarathy@chn.cts-corp.com 

Maturity  Level:  5 

Date  of  Appraisal:  Sept  2000 

Lead  Assessor(s):  V.  Kannan 


Computer  Sciences  Corporation  (CSC),  Aegis  Program 
Moorestown,  NJ 

Point  of  Contact:  Wendy  Irion  Talbot,  wirionta@csc.com 

Maturity  Level:  5 

Date  of  Appraisal:  March  2001 

Lead  Assessor(s):  Kathryn  Gallucci  (LE) 


Computer  Sciences  Corporation  (CSC),  Civil  Group 
Greenbelt,  MD 

Point  of  Contact:  Mel  Wahlberg,  mwahlber@csc.com 

Maturity  Level:  4 

Date  of  Appraisal:  Jan  2001 

Lead  Assessor(s):  Paul  Byrnes  (LA  &  LE) _ 

Computer  Sciences  Corporation  (CSC),  Civil  Group,  Systems,  Engineering,  and 
Analysis  Support  (SEAS)  Center 
Greenbelt,  MD 

Point  of  Contact:  Frank  McGarry,  fmcgarry@csc.com; 

Mel  Wahlberg,  mwahlber@csc.com 
Maturity  Level:  5 

Date  of  Appraisal:  Nov  1998  (SCE) 

Lead  Assessor(s):  Paul  Byrnes  (LA  &  LE) 
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High  M 

aturity  Organization 

Computer  Sciences  Corporation  (CSC),  Defense  Group  Aerospace  Information  Tech¬ 
nologies 

Dayton,  OH 

Point  of  Contact:  Cheryl  Plak,  cplak@csc.com 

Maturity  Level:  5 

Date  of  Appraisal:  Feb  1999  (SCE) 

Lead  Assessor(s):  Kathryn  Gallucci  (LE) 

Computer  Sciences  Corporation  (CSC),  Integrated  Systems  Division  (ISD) 

Moorestown,  NJ 

Point  of  Contact:  Bryan  Cooper,  bcooperl  @csc.com 

Maturity  Level:  4 

Date  of  Appraisal:  May  1998  (SCE) 

Lead  Assessor(s):  Paul  Byrnes  (LA  &  LE) 

Computer  Sciences  Corporation  (CSC),  Tactical  Systems  Center  (TSC) 

Moorestown,  NJ 

Point  of  Contact:  Wendy  Irion  Talbot,  wirionta@csc.com; 

Jeff  McGarry,  jmcgarrl  @csc.com 

Maturity  Level:  4 

Date  of  Appraisal:  May  1998  (SCE) 

Lead  Assessor(s):  Paul  Byrnes  (LA  &  LE) 

Covansys 

San  Francisco,  CA 

Point  of  Contact:  Prasanth  Kedarisetty,  KPrasanth@Covansys.com 

Maturity  Level:  4 

Date  of  Appraisal:  Jan  2001 

Lead  Assessor(s):  Richard  Knudson 

DCM  Technologies,  DCM  ASIC  Technology  Limited 

New  Delhi,  India 

Point  of  Contact:  Naresh  C.  Maheshwari,  ncm@dcmds.co.in 

Maturity  Level:  5 

Date  of  Appraisal:  April  2000 

Lead  Assessor(s):  Richard  Storch 

DSQ  Software 

Chennai,  India 

Point  of  Contact:  K.N.  Ananth,  kna@md.in.dsqsoft.com 

Maturity  Level:  4 

Date  of  Appraisal:  June  1998 

Lead  Assessor(s):  Judy  Bamberger 

Future  Software  Private  Limited 

Chennai,  India 

Point  of  Contact:  M.G.  Thomas,  thomasmg@future.futsoft.com 

Maturity  Level:  4 

Date  of  Appraisal:  June  1999 

Lead  Assessor(s):  Pradeep  Udhas 

HCL  Perot  Systems 

Noida  and  Bangalore,  India 

Point  of  Contact:  Rakesh  Soni,  rakesh.soni@hpsglobal.com 

Maturity  Level:  5 

Date  of  Appraisal:  Feb  2000 

Lead  Assessor(s):  Pradeep  Udhas 
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HighM 

aturity  Organization 

V 

HCL  Technologies  Limited,  Applications  Solutions  Development  Centre 

Chennai,  India 

Point  of  Contact:  N.  N.  Jha,  nnjha@msdc.hcltech.com 

Maturity  Level:  4 

Date  of  Appraisal:  May  2000 

Lead  Assessor(s):  V.  Kannan 

V 

HCL  Technologies  Limited,  Core  Technologies  Division 

Chennai,  India 

Point  of  Contact:  K.  R.  Gopinath,  krg@hclt.com 

Maturity  Level:  4 

Date  of  Appraisal:  Dec  2000 

Lead  Assessor(s):  Krishnamurthy  Kothandaraman  Raman 

HCL  Technologies  Limited,  Gurgaon  Software  Development  Center 

Gurgaon,  India 

Point  of  Contact:  Sanjeev  Gupta,  gsanjeev@ggn.hcltech.com 

Maturity  Level:  4 

Date  of  Appraisal:  July  2000 

Lead  Assessor(s):  V.  Kannan 

Hexaware  Technologies  Limited,  Mumbai  and  Chennai  Operations 

Chennai,  India 

Point  of  Contact:  Sulochana  Ganesan,  sulochana@hexaware.co.in 

Maturity  Level:  5 

Date  of  Appraisal:  Dec  2000 

Lead  Assessor(s):  V.  Kannan 

Honeywell  International,  Avionics  Integrated  Systems  (formerly  AlliedSignal,  Guid¬ 
ance  &  Control  Systems) 

Teterboro,  NJ 

Point  of  Contact:  Steve  Janiszewski,  stephen.janiszewski@honeywell.com 

Maturity  Level:  4 

Date  of  Appraisal:  Nov  1996 

Lead  Assessor(s):  Larry  Bramble  (I) 

Hughes  Software  Systems 

Bangalore  and  Gurgaon,  India 

Point  of  Contact:  Gautam  Brahma,  gbrahma@hss.hns.com 

Maturity  Level:  4 

Date  of  Appraisal:  Jan  2000 

Lead  Assessor(s):  V.  Kannan 

V 

IBM  Global  Services  India 

Bangalore,  India 

Point  of  Contact:  Asha  Goyal,  gasha@in.ibm.com; 

Maya  Srihari,  smaya@in.ibm.com 

Maturity  Level:  5 

Date  of  Appraisal:  Nov  1999 

Lead  Assessor(s):  Richard  Storch 

V 

i-flex  solutions  limited  (formerly  Citicorp  Information  Technology  Industries  Limited 
aka  CITIL) 

Bangalore,  India 

Point  of  Contact:  Vivek  V.  Govilkar,  vivek.govilkar@iflexsolutions.com 

Maturity  Level:  4 

Date  of  Appraisal:  Dec  1995 

Lead  Assessor(s):  Ken  Dymond 
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High  Maturity  Organization  | 

V 

i-flex  solutions  limited  (formerly  Citicorp  Information  Technology  Industries  Limited 
aka  CITIL) 

Mumbai,  India 

Point  of  Contact:  Vivek  Govilkar,  vivek.goviIkar@citicorp.com 

Maturity  Level:  4 

Date  of  Appraisal:  Dec  1995 

Lead  Assessor(s):  Cindi  Wise,  Ken  Dymond 

V 

i-flex  solutions  limited  Data  Warehouse  Center  of  Excellence 

Bangalore,  India 

Point  of  Contact:  Vivek  V.  Govilkar,  vivek.goviIkar@iflexsolutions.com 

Maturity  Level:  5 

Date  of  Appraisal:  Nov  1999 

Lead  Assessor(s):  Ken  Dymond,  Santhanakrishnan  Srinivasan,  Anand  Kumar 

i-flex  solutions  limited  IT  Services  Division 

Bangalore,  India 

Point  of  Contact;  Anand  Kumar,  anand.kumar@iflexsolutions.com 

Maturity  Level:  5 

Date  of  Appraisal:  Dec  2000 

Lead  Assessor(s);  Santhanakrishnan  Srinivasan,  Anand  Kumar 

i-flex  solutions  limited  IT  Services  Division,  Mumbai,  India 

Point  of  Contact:  Anand  Kumar,  anand.kumar@iflexsolutions.com 

Maturity  Level:  5 

Date  of  Appraisal:  Dec  2000 

Lead  Assessor(s):  Santhanakrishnan  Srinivasan,  Anand  Kumar,  Atul  Gupta 

Information  Technology  (India)  Ltd. 

Delhi,  India 

Point  of  Contact:  Madhumita  Poddar  Sen,  madhumitap@itil.com 

Maturity  Level:  4 

Date  of  Appraisal;  April  2000 

Lead  Assessor(s):  Pradeep  Udhas 

Intelligroup  Asia  Private  Limited,  Advanced  Development  Center 

Hyderabad,  India 

Point  of  Contact:  G.V.S.  Sharma,  gvs.sharma@intelligroup.co.in 

Maturity  Level:  5 

Date  of  Appraisal:  Oct  2000 

Lead  Assessor(s):  Raghav  S.  Nandyal,  John  Harding 

ITC  Infotech  India  Limited 

Bangalore,  India 

Point  of  Contact:  Paresh  Master,  pareshmaster@vsnl.com 

Maturity  Level:  5 

Date  of  Appraisal:  Aug  2000 

Lead  Assessor(s):  Richard  Storch 

Kshema  Technologies  Limited 

Bangalore,  India 

Point  of  Contact:  V.  Bhaskar,  vbhaskar@kshema.com 

Maturity  Level:  4 

Date  of  Appraisal:  March  2001 

Lead  Assessor(s):  Krishnamurthy  Kothanda  Raman 

L  &  T  Information  Technology  Limited 

Chennai,  India 

Point  of  Contact:  Anil  S.  Pandit,  anil.pandit@vashimail.ltitl.com 

Maturity  Level:  4 

Date  of  Appraisal:  Feb  2000 

Lead  Assessor(s):  V.  Kannan 
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High  M 

aturity  Organization 

Litton  Guidance  and  Control  Systems 

Woodland  Hills,  CA 

Point  of  Contact:  Roy  Nakahara,  nakaharr@littongcs.com 

Maturity  Level:  4 

Date  of  Appraisal:  Dec  1998 

Lead  Assessor(s):  Mark  Amaya 

V 

Litton/PRC  Inc. 

McLean,  VA  and  Colorado  Springs,  CO 

Point  of  Contact:  A1  Pflugrad,  pflugrad_al@prc.com 

Maturity  Level:  5 

Date  of  Appraisal:  March  2000  (SCE) 

Lead  Assessor(s):  Joseph  Morin  (LE) 

V 

Lockheed  Martin  Aeronautics  Company  (formerly  Lockheed  Martin  Tactical  Aircraft 
Systems  -  LMTAS) 

Fort  Worth,  TX 

Point  of  Contact:  Phil  Gould,  philip.c.gould@lmco.com 

Maturity  Level:  4 

Date  of  Appraisal:  Dec  1999 

Lead  Assessor(s):  Leia  Bowers  White 

Lockheed  Martin  Air  Traffic  Management 

Rockville,  MD 

Point  of  Contact:  Jim  Sandford,  jim.sandford@lmco.com 

Maturity  Level:  4 

Date  of  Appraisal:  Dec  1999 

Lead  Assessor(s):  Carol  Granger-Parker,  Jeff  Facemire 

Lockheed  Martin  Federal  Systems 

Owego,  NY 

Point  of  Contact:  Ed  Fontenot,  ed.fontenot@lmco.com; 

Warren  A.  Schwomeyer,  warren.schwomeyer@lmco.com 

Maturity  Level:  5 

Date  of  Appraisal:  Dec  1997 

Lead  Assessor(s):  John  Travalent,  Mary  Busby 

Lockheed  Martin  Information  Systems 

Orlando,  FL 

Point  of  Contact:  Michael  Ziomek,  michael.ziomek@lmco.com 

Maturity  Level:  4 

Date  of  Appraisal:  June  2000 

Lead  Assessor(s):  Gene  Jorgensen 

Lockheed  Martin  Management  &  Data  Systems 

King  of  Prussia,  PA 

Point  of  Contact:  M.  Lynn  Penn,  mary.lynn.penn@lmco.com 

Maturity  Level:  5 

Date  of  Appraisal:  Dec  2000 

Lead  Assessor(s):  Andy  Felschow,  Carol  Granger-Parker,  Dennis  Ring 

Lockheed  Martin  Mission  Systems 

Gaithersburg,  MD 

Point  of  Contact:  Paul  Weiler,  paul.weiler@lmco.com; 

A1  Aldrich,  al.aldrich@lmco.com 

Maturity  Level:  5 

Date  of  Appraisal:  Oct  1999  (SCE) 

Lead  Assessor (s):  Paul  Byrnes  (LA  &  LE) 
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HighM 

aturity  Organization 

Lockheed  Martin  Naval  Electronics  &  Surveillance  Systems  -  Syracuse 

Syracuse,  NY 

Point  of  Contact;  Peter  Barletto,  pete.barletto@lmco.com 

Maturity  Level:  5 

Date  of  Appraisal:  Nov  1999 

Lead  Assessor(s):  Carol  Granger-Parker,  Andy  Felschow 

Lockheed  Martin  Naval  Electronics  &  Surveillance  Systems-Eagan 

Eagan,  MN 

Point  of  Contact:  John  Travalent,  john.travalent@lmco.com 

Maturity  Level:  4 

Date  of  Appraisal:  Oct  1999 

Lead  Assessor(s):  Mary  Busby 

Lockheed  Martin  Naval  Electronics  &  Surveillance  Systems-Manassas  (formerly  Un¬ 
dersea  Systems) 

Manassas,  VA 

Point  of  Contact:  Dana  Roper,  dana.roper@Imco.com 

Maturity  Level:  5 

Date  of  Appraisal:  Feb  1999 

Lead  Assessor(s):  Judah  Mogilensky,  John  Travalent,  Donald  White 

Lockheed  Martin  Naval  Electronics  &  Surveillance  Systems-Moorestown 

Moorestown,  NJ 

Point  of  Contact:  Nghia  N.  Nguyen,  nghia.n.nguyen@lmco.com; 

Jeff  Tait,  Jeffery.a.tait@lmco.com 

Maturity  Level:  4 

Date  of  Appraisal:  Dec  1999 

Lead  Assessor(s):  Kevin  Schaan,  Kent  Johnson,  Dennis  Ring 

Lockheed  Martin  Space  Electronics  and  Communications  Systems-Manassas  (for¬ 
merly  Loral  Federal  Systems) 

Manassas,  VA 

Point  of  Contact:  Dana  Roper,  dana.roper@Imco.com 

Maturity  Level:  4 

Date  of  Appraisal:  June  1995 

Lead  Assessor(s):  Judah  Mogilensky,  John  Travalent,  Chris  Manak  (I) 

V 

Mastek  Limited 

Mumbai,  India 

Point  of  Contact:  P.  Rajshekharan,  rajshekhar@mastek.com 

Maturity  Level:  5 

Date  of  Appraisal:  Sept  2000 

Lead  Assessor(s):  Ron  Radice 

Motorola  Australia  Software  Centre 

Adelaide,  Australia 

Point  of  Contact:  Peter  Dew,  pdew@asc.corp.mot.com 

Maturity  Level:  4 

Date  of  Appraisal:  Aug  1997 

Lead  Assessor(s):  John  Pellegrin  (I) 

Motorola  China  Software  Center 

Beijing  &  Nanjing,  China 

Point  of  Contact:  John  Jun'an  Yu,  johny@sc.mcel.mot.com 

Maturity  Level:  5 

Date  of  Appraisal:  Sept  2000 

Lead  Assessor(s):  Dan  Weinberger,  Patricia  McNair 
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High  Maturity  Organization 

Motorola  India  Electronics  Ltd.  (MIEL) 

Bangalore,  India 

Point  of  Contact:  Sarala  Ravishankar,  sarala@miel.mot.comMaturity  Level:  5 

Date  of  Appraisal:  Nov  1993 

Lead  Assessor(s):  John  Pellegrin  (I) 

Motorola,  Asia  Pacific  Telecom  Carrier  Solutions  Group  (TCSG)  Applied  R&D  Cen¬ 
ter 

Beijing,  China 

Point  of  Contact:  Graham  Hu,  qchI422@emaiLmot.com 

Maturity  Level:  5 

Date  of  Appraisal:  Dec  2000  (CAE-compliant  Motorola  QSR  Subsystem  10  Software 
Assessment) 

Lead  Assessor(s):  (Fathi  Hakam  -  Motorola  Assessor) 

Motorola,  GSM  (Global  System  for  Mobile  Communications)  Systems  Division,  Net¬ 
work  Systems  Group 

Arlington  Heights,  IL 

Point  of  Contact:  Barbara  Hirsh,  hirsh@cig.mot.com 

Maturity  Level:  5 

Date  of  Appraisal:  Oct  1997  (CAF-compliant  Motorola  QSR  Subsystem  10  Software 
Assessment) 

Lead  Assessor(s):  (Ellen  Pickthall  --  Motorola  Assessor) 

NCR  Corporation,  Teradata  Development  Division,  Massively  Parallel  Systems 

San  Diego,  CA 

Point  of  Contact:  Ron  Weidemann,  ron.weidemann@sandiegoca.ncr.com 

Maturity  Level:  4 

Date  of  Appraisal:  Oct  1999 

Lead  Assessor(s):  Ron  Weidemann 

Network  Systems  and  Technologies  (P)  Ltd 

Trivandrum,  India 

Point  of  Contact:  S  K  Pillai,  skp@nestec.net 

Maturity  Level:  5 

Date  of  Appraisal:  May  2000 

Lead  Assessor(s):  Ron  Radice 

NUT  Limited 

New  Delhi,  India 

Point  of  Contact:  Bhaskar  Chavali,  BhaskarC@niit.com 

Maturity  Level:  5 

Date  of  Appraisal:  Sept  1999 

Lead  Assessor(s):  Richard  Storch 

Northrop  Grumman  Electronic  Sensors  and  Systems  Sector  (ESSS) 

Baltimore,  MD 

Point  of  Contact:  Eva  M.  Brandt,  eva_m_brandt@md.northgrum.com 

Maturity  Level:  4 

Date  of  Appraisal:  Oct  1999 

Lead  Assessor(s):  John  Blyskal 

Northrop  Grumman,  Air  Combat  Systems,  Integrated  Systems  and  Aeronautics  Sector 

El  Segundo,  CA 

Point  of  Contact:  Leitha  Purcell,  purcele@mail.northgrum.com 

Maturity  Level:  4 

Date  of  Appraisal:  Oct  1998 

Lead  Assessor(s):  Don  Dortenzo 
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High  Maturity  Organization  I 

Northrop  Grumman,  Integrated  Systems  &  Aerostructures,  AEW  &  EW  Systems 
(formerly  Surveillance  &  Battle  Management) 

Bethpage,  NY 

Point  of  Contact:  Dennis  Carter,  cartede@mail.northgrum.com 

Maturity  Level:  4 

Date  of  Appraisal:  Oct  1998 

Lead  Assessor(s):  Andy  Felschow 

Oracle  Software  India  Limited,  India  Development  Center 

Bangalore,  India 

Point  of  Contact:  Ashish  Saigal,  asaigal@in.oracle.com 

Maturity  Level:  4 

Date  of  Appraisal:  May  1999 

Lead  Assessor(s):  Pradeep  Udhas 

Patni  Computer  Systems  Ltd.  (PCS),  Mumbai,  Navi  Mumbai,  Pune  and  Gandhinagar 
Facilities 

Mumbai,  India 

Point  of  Contact:  Sunil  Kuwalekar,  sunil.kuwalekar@patni.com; 

N  A  Nagwekar,  nilendra.nagwekar@patni.com 

Maturity  Level:  5 

Date  of  Appraisal:  Aug  2000 

Lead  Assessor(s):  Pradeep  Udhas 

V 

Philips  Consumer  Electronics,  Philips  Software  Centre 

Bangalore,  India 

Point  of  Contact:  Bob  Hoekstra,  bob.hoekstra@philips.com 

Maturity  Level:  5 

Date  of  Appraisal:  July  2000 

Lead  Assessor(s):  Richard  Knudson 

Raytheon  (formerly  Raytheon  E-Systems) 

Garland,  TX 

Point  of  Contact:  Mary  E.  Howard,  mary_e_howard@raytheon.com 

Maturity  Level:  4 

Date  of  Appraisal:  Dec  1998 

Lead  Assessor(s):  Neil  Potter 

Raytheon  C3I  Fullerton  Integrated  Systems,  Command  and  Control  Systems/Middle 

East  Operations 

Fullerton,  CA 

Point  of  Contact:  Jane  A.  Moon,  jmoon@west.raytheon.com; 

Janet  Bratton,  Jabratton@west.raytheon.com 

Maturity  Level:  5 

Date  of  Appraisal:  Oct  1998 

Lead  Assessor(s):  Paul  Byrnes  (LA  &  LE),  Jane  Moon,  Ronald  Ulrich,  Ivan  Flinn, 

Bruce  Duncil  (LA  &  LE),  Janet  Bratton 

Raytheon  Missile  Systems,  Software  Engineering  Center 

Tucson,  AZ 

Point  of  Contact:  Michael  D.  Scott,  mscottl  @  west.raytheon.com 

Maturity  Level:  4 

Date  of  Appraisal:  Oct  1998 

Lead  Assessor(s):  John  Ryskowski,  Michael  Scott 

Raytheon,  Electronic  Systems,  Sensors  Engineering 

El  Segundo,  CA 

Point  of  Contact:  Paul  Curry,  pcurry@west.raytheon.com 

Maturity  Level:  4 

Date  of  Appraisal:  Oct  2000 

Lead  Assessor(s):  Janet  Bratton,  Michael  Scott,  Ivan  Flinn 
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High  Maturity  Organization 

Satyam  Computer  Services  Ltd 

India 

Point  of  Contact:  Prabhuu  Sinha,  prabhuu@satyam.com 

Maturity  Level:  5 

Date  of  Appraisal:  March  1999 

Lead  Assessor(s):  Richard  Knudson 

Siemens  Information  Systems  Limited  (SISL),  Software  Development  Group 

Bangalore,  India 

Point  of  Contact:  T.  Kathavarayan,  kathavarayan.t@sisLco.in 

Maturity  Level:  4 

Date  of  Appraisal:  Aug  2000 

Lead  Assessor(s):  Richard  Storch 

Silverline  Technologies  Limited 

Mumbai,  India 

Point  of  Contact:  S.  Purushotham,  sp@silverline.com 

Maturity  Level:  4 

Date  of  Appraisal:  Dec  1999 

Lead  Assessor(s):  V.  Kannan 

V 

Tata  Consultancy  Services 

Ahmedabad,  India 

Point  of  Contact:  Gargi  Keeni,  gkeeni@mumbai.tcs.co.in; 

Rosemary  Hedge,  rhedge@ahd.tcs.co.in 

Maturity  Level:  5 

Date  of  Appraisal:  Nov  2000 

Lead  Assessor (s):  Ron  Radice,  P.  Suresh 

V 

Tata  Consultancy  Services,  Ambattur,  Chennai,  India 

Point  of  Contact:  Gargi  Keeni,  gkeeni@mumbai.tcs.co.in 

Maturity  Level:  5 

Date  of  Appraisal:  July  2000 

Lead  Assessor(s):  Ron  Radice 

V 

Tata  Consultancy  Services 

Bangalore,  India 

Point  of  Contact:  Gargi  Keeni,  gkeeni@mumbai.tcs.co.in; 

Uma  Rijhwani,  umarijhwani@bIore.tcs.co.in 

Maturity  Level:  5 

Date  of  Appraisal:  Jan  2000 

Lead  Assessor(s):  Ron  Radice 

V 

Tata  Consultancy  Services 

Calcutta,  India 

Point  of  Contact:  Gargi  Keeni,  gkeeni@mumbai.tcs.co.in; 

Arunava  Chandra,  achandra@tcscaLco.in 

Maturity  Level:  5 

Date  of  Appraisal:  Jan  2000 

Lead  Assessor(s):  Ron  Radice 

V 

Tata  Consultancy  Services,  Global  Engineering  Development  Center 

Chennai,  India 

Point  of  Contact:  Gargi  Keeni,  gkeeni@mumbai.tcs.co.in; 

M.  Mala,  mala@wst03.tata.ge.com 

Maturity  Level:  5 

Date  of  Appraisal:  July  2000 

Lead  Assessor(s):  John  Harding 
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aturity  Organization 

V 

Tata  Consultancy  Services,  Gurgaon  II 

New  Delhi,  India 

Point  of  Contact:  Gargi  Keeni,  gkeeni@mumbai.tcs.co.in 

Maturity  Level:  5 

Date  of  Appraisal:  Feb  2001 

Lead  Assessor(s):  Ron  Radice 

V 

Tata  Consultancy  Services,  HP  Centre 

Chennai,  India 

Point  of  Contact:  Gargi  Keeni,  gkeeni@niumbai.tcs.co.in; 

P.  Vasu,  pvasu@hp.india.com 

Maturity  Level:  5 

Date  of  Appraisal:  July  1999 

Lead  Assessor(s):  Ron  Radice 

V 

Tata  Consultancy  Services 

Hyderabad,  India 

Point  of  Contact:  Gargi  Keeni,  gkeeni@mumbai.tcs.co.in; 

N  V  Jayaramakrishna,  jayaram@hydbad.tcs.co.in 

Maturity  Level:  5 

Date  of  Appraisal:  May  2000 

Lead  Assessor(s):  John  Harding,  Gargi  Keeni 

V 

Tata  Consultancy  Services 

Lucknow,  India 

Point  of  Contact:  Gargi  Keeni,  gkeeni@mumbai.tcs.co.in; 

Nirmal  Kumar,  nirmal_kumar@lko.tcs.co.in 

Maturity  Level:  5 

Date  of  Appraisal:  Jan  2000 

Lead  Assessor(s):  John  Harding,  Radhika  Sokhi 

V 

Tata  Consultancy  Services,  SEEPZ 

Mumbai,  India 

Point  of  Contact:  Gargi  Keeni,  gkeeni@mumbai.tcs.co.in; 

P.  Suresh,  p.suresh@seepz.tcs.co.in 

Maturity  Level:  5 

Date  of  Appraisal:  Aug  1999 

Lead  Assessor(s):  Ron  Radice 

V 

Tata  Consultancy  Services 

Shollinganallur,  Chennai,  India 

Point  of  Contact:  Gargi  Keeni,  gkeeni@mumbai.tcs.co.in; 

R.  Ravishankar,  rravisha@chennai.tcs.co.in 

Maturity  Level:  5 

Date  of  Appraisal:  Nov  1999 

Lead  Assessor(s):  Ron  Radice 

V 

Tata  Consultancy  Services,  US  West 

Chennai,  India 

Point  of  Contact:  Gargi  Keeni,  gkeeni@mumbai.tcs.co.in; 

R.  Umasankar,  rumasan@uswest.com 

Maturity  Level:  5 

Date  of  Appraisal:  April  1999 

Lead  Assessor(s):  Ron  Radice,  V.  Muralidharan,  John  Harding 

Tata  Elxsi  Limited 

Bangalore,  India 

Point  of  Contact:  M.  Thangarajan,  mtr@teil.soft.net 

Maturity  Level:  4 

Date  of  Appraisal:  Aug  1999 

Lead  Assessor(s):  Pradeep  Udhas 
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High  Maturity  Organization 

Telcordia  Technologies,  Inc. 

Morristown,  NJ 

Point  of  Contact:  Bill  Pitterman,  wpitterm@telcordia.com 

Maturity  Level:  5 

Date  of  Appraisal:  May  1999 

Lead  Assessor(s):  Pat  O’Toole,  Bill  Curtis,  Norm  Hammock 

V 

U.S.  Air  Force,  Ogden  Air  Logistics  Center,  Technology  &  Industrial  Support  Direc¬ 
torate,  Software  Engineering  Division 

Hill  AFB,  UT 

Point  of  Contact:  Jim  Vanfleet,  Jim.Vanfleet@Hill.af.mil 

Maturity  Level:  5 

Date  of  Appraisal:  July  1998 

Lead  Assessor(s):  Mark  Paulk,  Brian  Larman,  Donna  Dunaway,  Bonnie  Bollinger, 

Millie  Sapp,  Mike  Ballard 

V 

U.S.  Air  Force,  Oklahoma  City  Air  Logistics  Center,  Directorate  of  Aircraft  Manage¬ 
ment,  Software  Division,  Test  Software  and  Industrial  Automation  Branches  (OC- 
ALC/LAS) 

Tinker  AFB,  OK 

Point  of  Contact:  Kelley  Butler,  kelley.butler@tinker.af.mil 

Maturity  Level:  4 

Date  of  Appraisal:  Nov  1996 

Lead  Assessor(s):  Judah  Mogilensky 

U.S.  Army  Aviation  &  Missile  Command,  Software  Engineering  Directorate 

Redstone  Arsenal,  AL 

Point  of  Contact:  Jacquelyn  Langhout,  jackie.langhout@sed.redstone.army.mil 

Maturity  Level:  4 

Date  of  Appraisal:  April  2000 

Lead  Assessor(s):  David  Zubrow 

V 

U.S.  Army,  Communications  and  Electronics  Command  (CECOM),  Software  Engi¬ 
neering  Center  (SEC),  Fire  Support  Software  Engineering  (Telos) 

Fort  Sill,  OK 

Point  of  Contact:  Don  Couch,  couchdc@fssec.army.mil; 

Phil  Sperling,  sperlips@fssec, army. mil 

Maturity  Level:  4 

Date  of  Appraisal:  Nov  1997 

Lead  Assessor(s):  Don  Couch,  David  Zubrow 

U.S.  Navy,  F/A-18  Software  Development  Task  Team  (SWDTT),  Naval  Air  Warfare 
Center  Weapons  Division  (NAWCWD) 

China  Lake,  CA 

Point  of  Contact:  Claire  Velicer,  velicercm@navair.navy.mil 

Maturity  Level:  4 

Date  of  Appraisal:  Feb  2000 

Lead  Assessor(s):  Tim  Olson,  Ralph  Williams 

U.S.  Navy,  Fleet  Material  Support  Office 

Mechanicsburg,  PA 

Point  of  Contact:  Kathleen  D.  Chastain,  kathleen_chastain@fmso.navy.mil 

Maturity  Level:  4 

Date  of  Appraisal:  Oct  1998 

Lead  Assessor(s):  John  Smith,  Ann  Roberts 

V 

United  Space  Alliance,  Space  Shuttle  Onboard  Software  Project 

Houston,  TX 

Point  of  Contact:  Julie  Barnard,  julie.r.barnard@usahq.unitedspacealliance.com 

Maturity  Level:  5 

Date  of  Appraisal:  Nov  1989  (SCE) 

Lead  Assessor(s):  Donald  Sova  (before  LA  and  LE  programs) 

CMU/SEI-2001-SR-014 


205 


High  M 

aturity  Organization 

Wipro  GE  Medical  Systems 

Bangalore,  India 

Point  of  Contact:  K.  Puhazhendi,  k.puhazhendi@geind.ge.com 

Maturity  Level:  5 

Date  of  Appraisal:  Jan  1999 

Lead  Assessor(s):  Richard  Knudson,  C.  Rama  Rao 

Wipro  Technologies,  Enterprise  Solutions  Division 

Bangalore,  India 

Point  of  Contact:  T.  V.  Subbarao,  subbarao.tangirala@wipro.com 

Maturity  Level:  5 

Date  of  Appraisal:  Dec  1998 

Lead  Assessor(s):  Richard  Storch 

Wipro  Technologies,  Global  R  &  D  (formerly  Technology  Solutions) 

Bangalore,  India 

Point  of  Contact:  V.  Subramanyam,  vsm@wipinfo.soft.net 

Maturity  Level:  5 

Date  of  Appraisal:  June  1999 

Lead  Assessor(s):  Richard  Knudson,  Mark  Paulk 

V 

Zensar  Technologies  Limited  (formerly  International  Computers  India  Limited) 

Pune, India 

Point  of  Contact:  Ashok  Sontakke,  a.r.sontakke@icil.co.in 

Maturity  Level:  5 

Date  of  Appraisal:  Feb  1999 

Lead  Assessor(s):  Richard  Knudson 
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Acronyms 


ANSI 

ASQ 

ASQC 

CAF 

CBA IPI 

CCB 

CM 

CM 

CMM 

CMMI 

CMU 

COQ 

DoD 

DP 

EIA 

EITVOX 

ETVX 

FFRDC 


American  National  Standards  Institute 
American  Society  for  Quality  (formerly  ASQC) 

American  Society  for  Quality  Control 
CMM  Appraisal  Framework 

CMM-based  appraisal  for  internal  process  improvement 
configuration  control  board 

[Software]  Configuration  Management  (Software  CMM  Level  2 
KPA) 

configuration  management 
capability  maturity  model 
CMM  Integration 
Carnegie  Mellon  University 
cost  of  quality 
Department  of  Defense 

Defect  Prevention  (Software  CMM  Level  5  KPA) 

Electronics  Industries  Association 

entry  criteria,  inputs,  task,  verification,  outputs,  and  exit  criteria 
entry  criteria,  task,  verification,  and  exit  criteria 
federally  funded  research  and  development  center 
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FTE 

HMO 

HTI 

1C 

IDEFO 

lEC 

IEEE 

IM 

ISM 

ISO 

IT 

KPA 

KSLOC 

LA 

LE 

LOC 

Mod 

NSIA 

OPD 

OPF 

OSD 


full-time  equivalent 
high  maturity  organization 
happy  team  index 

Intergroup  Coordination  (Software  CMM  Level  3  KPA) 
function  modeling  method 
International  Electrotechnical  Commission 
Institute  of  Electrical  and  Electronic  Engineers,  Inc. 

Integrated  [Software]  Management  (Software  CMM  Level  3  KPA) 

Integrated  Software  Management  (Software  CMM  Level  3  KPA) 

International  Organization  for  Standardization 

information  technology 

key  process  area 

1000  source  lines  of  code 

Lead  Assessor 

Lead  Evaluator 

lines  of  code 

Ministry  of  Defence  (United  Kingdom) 

National  Security  Industrial  Association 

Organization  Process  Definition  (Software  CMM  Level  3  KPA) 

Organization  Process  Focus  (Software  CMM  Level  3  KPA) 

Office  of  the  Secretary  of  Defense 
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PAL 


process  asset  library 


PCM 

PD 

PE 

PF 

PM 

PP 

PR 

PSP 

PT 

PTO 

QA 

QA 

QFD 

QM 

QMS 

QP 

QPM 

RM 

ROI 


Process  Change  Management  (Software  CMM  Level  5  KPA) 
[Organization]  Process  Definition  (Software  CMM  Level  3  KPA) 
[Software]  Product  Engineering  (Software  CMM  Level  3  KPA) 
[Organization]  Process  Focus  (Software  CMM  Level  3  KPA) 
Process  [Change]  Management  (Software  CMM  Level  5  KPA) 
[Software]  Project  Planning  (Software  CMM  Level  2  KPA) 

Peer  Reviews  (Software  CMM  Level  3  KPA) 

Personal  Software  Process 

[Software]  Project  Tracking  [and  Oversight]  (Software  CMM 
Level  2  KPA) 

[Software]  Project  Tracking  and  Oversight  (Software  CMM  Level 
2  KPA) 

[Software]  Quality  Assurance  (Software  CMM  Level  2  KPA) 

quality  assurance 

quality  function  deployment 

[Software]  Quality  Management  (Software  CMM  Level  4  KPA) 
quality  management  system 

Quantitative  Process  [Management]  (Software  CMM  Level  4 
KPA) 

Quantitative  Process  Management  (Software  CMM  Level  4  KPA) 
Requirements  Management  (Software  CMM  Level  2  KPA) 
return  on  investment 
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SADT 

SCAMPI 

SCE 

SCM 

SCM 

SEI 

SEPG 

SLOC 

SM 

SME 

SOW 

SPA 

SPE 

SPIN 

SPP 

SQA 

SQA 

SQM 

SSM 

TCM 


structured  analysis  and  design  technique 

standard  CMMI  assessment  method  for  process  improvement 

software  capability  evaluation  (method) 

software  configuration  management 

Software  Configuration  Management  (Software  CMM  Level  2 
KPA) 

Software  Engineering  Institute 
Software  Engineering  Process  Group 
source  lines  of  code 

[Software]  Subcontract  Management  (Software  CMM  Level  2 
KPA) 

subject  matter  expert 
statement  of  work 

software  process  assessment  (method) 

Software  Product  Engineering  (Software  CMM  Level  3  KPA) 

Software  Process  Improvement  Network 

Software  Project  Planning  (Software  CMM  Level  2  KPA) 

Software  Quality  Assurance  (Software  CMM  Level  2  KPA) 
software  quality  assurance 

Software  Quality  Management  (Software  CMM  Level  4  KPA) 
Software  Subcontract  Management  (Software  CMM  Level  2  KPA) 
Technology  Change  Management  (Software  CMM  Level  5  KPA) 


210 


CMU/SEI-2001-SR-014 


TM 


Technology  [Change]  Management  (Software  CMM  Level  5  KPA) 


TP 

Training  Program  (Software  CMM  Level  3  KPA) 

TQM 

Total  Quality  Management 

TSP 

Team  Software  Process 

USAF 

United  States  Air  Force 

WBS 

work  breakdown  structure 

WG 

working  group 
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